Meta's engineering team completed a staged migration that moved thousands of MySQL — based ingestion jobs from fragmented, pipeline — owned systems into a centralized, self-managed warehouse and retired the legacy pipelines. The platform involved is petabyte — scale and moves several petabytes of social — graph data per day; it feeds analytics, reporting, machine learning, and internal product development, so preserving consistency and availability was imperative.
The transition followed a three — stage deployment pattern. First, a shadow phase ran the new warehouse against live data to validate behavior without impacting production. Next, a reverse — shadow phase swapped production ownership to the warehouse while retaining rollback capability. Finally, a cleanup phase retired the legacy pipelines once all validation and operational checks passed.
To support correctness at scale, the team kept a change data capture (CDC) model for each job with three concrete artifacts: an internal full-dump table for initial loads, a delta table for ongoing incremental changes, and a target table consumed by downstream data customers. A central management service tracked job entities, table names and schemas so migrations could be validated and schema consistency maintained across thousands of jobs during cutover.
Validation relied on continuous, automated checks across those artifacts. Engineers compared row counts and checksums between production and shadow jobs, investigated mismatches promptly, and re-ran fixes in pre-production before proceeding. The team also measured compute and storage quotas for shadow jobs to ensure the production environment had sufficient resources before advancing to ownership swap; as Zihao Tao and colleagues explained, "We continuously monitored row count and checksum mismatches... We also measured the compute and storage quotas for the shadow jobs to ensure that the production environment had sufficient resources before proceeding."
Because CDC required expensive full snapshots for initial loads and for recovery after fixes, the engineers limited the creation of shadow jobs until data quality issues were resolved to avoid repeated large — scale full dumps. They imposed strict correctness and performance gates — comparing counts and checksums, monitoring latency and resource consumption — and added extra requirements for critical tables that dependent teams relied on. Operational safeguards included automated rollback controls and lifecycle tracking for every migration job so teams could revert a cutover if checks failed. The migration also introduced compatibility layers to smooth schema transitions and maintain client compatibility during the swap.
Sources
Replies (0)
No replies in this topic yet.
