The Etsy engineering team recently described how the company migrated its long-running MySQL sharding infrastructure to Vitess. The transition moved shard routing from Etsy’s internal systems to Vitess using vindexes, enabling capabilities such as resharding data and sharding previously unsharded tables.
An open source database clustering system for horizontally scaling MySQL, Vitess was initially introduced as a layer between the ORM (Object-Relational Mapping) and the database, routing queries through it. At the same time, the ORM continued to specify the target shard.
In Vitess, vindexes define how application data maps to database shards and how queries are routed across them. Ella Yarmo-Gray, senior software engineer at Etsy, explains the challenge Etsy faced:
With this new infrastructure in place, we were ready to start exploring vindexes, which define sharding strategies within Vitess (…) Since the ORM’s shard mappings are random and not algorithmic, using one of these out-of-the-box would require re-sharding all of our data – a process that would be manual and likely take years. Instead, we chose to write custom vindexes that ported our existing shard logic into Vitess so we could test how vindexes worked in our environment without the complexity and risk of moving data.
Etsy has used a sharded MySQL architecture since around 2010 to store most of its production data, with proprietary sharding logic, approximately 1000 shards, and 425 TB of data, serving 1.7 million requests per second.
Etsy engineers access MySQL through an internal object-relational mapping (ORM) layer, in which each database table has a corresponding model. For sharded tables, a unique ID field, called the shardifier ID, was defined in the model to determine which shard stores each record: while most models use shop_id or user_id as a sharding key, overall more than 30 different IDs were used, with record-to-shard mappings stored in a single (unsharded) “index” database. While the approach improved scalability and limited the impact of outages to a small portion of traffic, scaling operations were slow and manual, with the index database being a single point of failure and developers having to manage sharding themselves.
A few years ago, the company decided to migrate to Vitess to address those challenges while maintaining MySQL compatibility, removing the “index” database, and hiding shard complexity from developers. The migration involved redesigning parts of the data model to support better sharding, selecting a shard key, and gradually moving production traffic to the new environment while verifying data consistency. Yarmo-Gray concludes:
Five years, approximately 2,500 pull requests and 6,000 queries later, we have successfully migrated Etsy’s shard management to Vitess vindexes! Despite the work we put in to streamline the migration process, it was still a challenge to replace the database infrastructure for a codebase of Etsy’s scale and age.
In the last few years, the Etsy engineering team has published a series of articles titled “Sharding Payments with Vitess,” documenting their migration effort for the payments platform, describing the challenges of migrating their data models, covering the cut-over effort of a crucial high-traffic system, and evaluating cutover risks.
