Sharding

The wall replication hits

Replication is great for reads. Add more replicas. Reads scale in a straight line. But every write still goes to a single primary. If your write rate is more than one machine can take, you are stuck.

Twitter does about 500 million tweets per day. That works out to about 6,000 tweets per second. No single database can absorb that write rate.

The fix is to shard the data. You split it across many independent databases. Each one holds a slice of the total. Each one handles its own writes. Your total write capacity is the number of shards times the per-shard limit.

How sharding works

Pick a shard key. That is a column whose value decides which shard a row belongs to. Usually it is user_id.

The rule looks like this. shard_index = hash(user_id) % num_shards.

User 42 hashes to shard 3. All of user 42's tweets, photos, and follows live on shard 3. User 999 hashes to shard 7. All of their data lives there.

When the app reads or writes for user 42, it computes the shard. It sends the query only to that shard. Each shard is its own independent database. Its own primary. Its own replicas. Its own everything.

Now you have N independent databases. Each handles 1 of every N requests. More shards means more capacity.

The hard parts

Sharding sounds simple but it brings real pain.

Cross-shard queries hurt. "Show me the top 10 users by post count" means querying every shard and merging the results in the app. There are no easy joins across shards.

Hot shards. If your shard key creates uneven load, one shard gets too much. One celebrity user with 100x the traffic can overload a shard while the others sit idle.

Resharding is expensive. Going from 4 shards to 8 shards means moving half your data. That is hours or days of work. Consistent hashing (the next concept) helps reduce how much data has to move.

Cross-shard transactions need a special protocol. They are slow and complex.

This is why most teams avoid sharding until they have used up every other option. It is the nuclear option.

When to shard

Signs you might need to shard.

Your primary database is maxed out on writes even after you tuned the CPU, the disk, and the query plans. You have already scaled the machine to the biggest size your cloud provider sells. Your dataset is too big to fit on one machine. Your replicas cannot keep up because the replication stream is too heavy.

The order to try things is this. Scale up the box first. Then add replicas for reads. Only shard when both of those run out. Many companies built apps with a billion users without sharding. They kept writes simple and pushed reads into replicas and caches.

Sharding is real and sometimes needed. But it is a heavy operational tax. Do not reach for it early.