Scaling AWS RDS

Hopefully this saves you 4 hours of time

So I revently wrote about scaling infrastructure.

Today I just want a quick memo on how to actually scale that damn thing.

In particular how to scale Amazon web service's (AWS) relational database service (RDS).

It's surprisingly how stressful clicking a few buttons can be when you don't know if it's going to cause the site to go offline for the millions of requests coming in.

Since no one knew what we were doing, it took us three engineers around 4 hrs to finally click a few buttons after trying to relieve the RDS from load unsuccessfully via other safer means.

So here goes:

• Make sure to have multi zone deployments enabled for your instance.

• Have at least a reader and writer instance (this should automatically be true if you have multi zone deployments.

• Update one of the reader instance to the desired size and set to tier-0 priority (top priority).

• Update all the other reader instances to the desired sizes.

• Manually force the main writer instance to failover. This should force the reader instance with the highest priority to become the writer instance. Happens within a 30 seconds.

• Manually upgrade the writer turned reader instance to the desired size.

• Profit.

Note, you might have to restart your service just to prune all the connections properly (otherwise some connections might be stuck trying to writer to the old writer instance, causing errors).

And that's it! You'll probably see some failed requests during that time, but the overall service should be healthy.