Swapping Low Latency Data Storage Under High Load

Miro's Redis instance couldn't scale for 700 engineers. Learn how they moved 140 million accounts to DynamoDB while handling 100,000 requests per second.

#1about 4 minutes

Migrating 140M users from a Redis-based monolith

Miro's user registry team faced migrating 140 million users from Redis, which was unsuitable as a persistent store and was tied to a difficult-to-maintain monolith.

#2about 3 minutes

Evaluating PostgreSQL vs DynamoDB for the new database

The team compared PostgreSQL's flexibility with DynamoDB's managed scaling, ultimately choosing DynamoDB paired with DAX for its performance on read-heavy workloads.

#3about 2 minutes

Load testing the database solutions under production load

To validate their choice, they load-tested PostgreSQL using pgbench and DynamoDB with a k6 and EC2 setup to simulate production traffic.

#4about 2 minutes

Understanding the limitations of DynamoDB and DAX

Adopting DynamoDB with DAX required navigating limitations like primary key access patterns, separate item and query caches, and difficult vertical scaling.

#5about 4 minutes

The phased migration strategy from Redis to DynamoDB

The migration was executed in controlled phases using feature flags, including dual writes, a background data copy, shadow calls for validation, and a gradual rollout.

#6about 2 minutes

Unexpected DAX performance issues and learnings

After going live, the team discovered that rapid connection spikes caused DAX throttling and that adding a fourth node to their cluster unexpectedly increased replication lag.

#7about 4 minutes

Results of the migration and Q&A

The successful migration reduced endpoint latency up to four times and enabled the team to move forward with their microservice architecture, followed by a Q&A on alternative solutions.