← Back to blog

The Most Boring Migration in History

At 8am on a Saturday morning, a 10 year old system migrated from its home in an outdated ec2 linux box holding every single component to a modern containerized stack and managed database instance. The CEO didn't even realize the migration had taken place, until he was alerted on Monday morning. The migration caused a grand total of about 60 minutes of degraded performance and no user complaints.

That is usually a quarter of work for five people. It was one outside contractor, part time, over eight weeks, most of which was spent waiting and running QA. Here is how it was done, and why you should consider the same approach.

What was on the server

Start with what that box was holding. The application, the database, every uploaded file, and the automated backups, all on one disk. Tens of gigabytes of database backups sat on the same volume as the database they were backing up, so losing that disk would have taken the data and its on-disk copies together.

The nightly backup script was truncated in the middle of an if statement. It died on every run with a syntax error and never produced a file in years. A second, complete script ran in the morning, so the backups directory always looked fresh, and that is exactly why nobody caught it. Whatever restore capability the company actually had did not come from that schedule.

The machine had one alarm. It checked disk usage by filtering df output for a device name that instance did not have. The filter returned nothing, the script threw before it ever evaluated the threshold, and the disk sat at 68 percent while an alert configured to fire at 80 percent was structurally incapable of firing.

The queue read an environment variable that nothing set anymore, so registration and password reset mail went out inline, inside the user's own HTTP request, while a queue worker sat next to it processing nothing. A single request to the sitemap pulled seventeen megabytes out of the database to return a page under one megabyte, across 848 queries.

Deployment was: log into the machine, pull the branch, restart the runtime. No CI, no infrastructure written down anywhere, no way back.

What was readable from the public internet

None of this was hidden. Anyone with access could have written that list in an afternoon, and a good part of it was readable from the public internet with nothing but a browser. A full server configuration dump, served publicly, listing runtime version, internal address and install path. An endpoint returning business metrics in the clear with no authentication, explicitly blocked in robots.txt, so the sensitivity was known and the endpoint was open anyway. Analytics running through a product that had been switched off three years earlier, so three years of not knowing where anyone came from.

What the setup cost the business

The business cost of all this is not the part people expect. The team was still shipping. Features went out at roughly the rate anyone would expect. What the setup cost was predictability. Nobody could tell in advance which change was safe, because there was no way to check before it was live and no way back if it wasn't. The biggest cost of working like this is risk, carried permanently, on every single release. Risk that size shapes what anyone is willing to attempt. The smallest change is always the safest one, so it wins every time, and after ten years the codebase is a decade of picking the smallest option.

Building the replacement beside production

So we did not migrate it. We built the replacement beside it and left the old machine as real production from the first day to the last. That one decision is why the risk stayed low and why a single part time person could carry it. Nothing we built could break the business until the moment we chose to switch.

The new stack runs in containers against a managed database with automated backups that actually run and deletion protection turned on. Uploads moved to object storage behind a CDN. Secrets came out of the repository. Deploy and rollback became a button. The infrastructure is all code, and it ran locally before a dollar of hosting was spent.

The bulk of the eight weeks went into a full replica of production, kept current by a one-way data sync, run and re-run for weeks. We broke that sync script on purpose, every way we could think of, which is how we found that a dying database dump on the left of a pipe reports the exit status of whatever is on the right. The obvious version of that script would have wiped the target and reported success. This is the part that looks like idle time on an invoice, and it is the reason the cutover came down to four checklist items instead of a weekend.

Monitoring got rebuilt around the lesson from the old box. An hourly check that asserts things are genuinely running, paired with an external heartbeat that fires if the checker itself goes quiet. An alarm that only fires on a threshold cannot tell you when it has stopped working.

The cutover

This brings me back to the cutover date, at 8am on a Saturday morning. With a replicated system fully up and running, we had a big checklist of items ahead of us for the cutover. We

  • Provisioned the new TLS certificates for the new servers ahead of the DNS cutover
  • Ran the sync script in parallel one last time, to ingest any data and content since the last run
  • Smoke checked the new instance one more time
  • Pointed the DNS at the new instance

And that's it. One hand is enough to count the steps.

None of those four steps is where the 60 minutes went. The site never went down. It got slow, for about an hour, from two problems that only exist under real traffic and that no replica can show you. The new database was provisioned too small, and an internal service degraded once it was carrying the whole load. One was fixed later that day. The other on Monday, when weekday load arrived and made it obvious. Migrations without unforeseen incidents simply don't exist. The work is keeping the incidents small enough that nobody files a ticket.

What a legacy migration buys, and what it does not

Eight weeks from the first audit to the cutover, about a month of that spent actually building. It cost less than a month of a consultant in a nice suit. The quarter with five FTEs would not have bought a better outcome, because what makes this safe is sequencing, not headcount. You cannot roast a turkey five times faster by using five ovens.

Here is what that buys and what it does not. A change can now be made, reversed, and observed. Backups run without anyone watching them. A deploy is a button, and the team knows exactly what went out, when, and who sent it, which nobody could know while everyone shared one set of SSH credentials. Nothing gets edited on a live server and pushed to main afterwards. That is the floor. It does not make the product better on its own, it makes the next round of work safe to do, and that work is separate.

If you hold a piece of a company running on something like this, the reason to consider the same approach is not the eight weeks. We cannot tell you your number without seeing your system, and anyone who quotes you one before looking is guessing. The reason is that building beside the old system fails safely. If the replacement is wrong, you find out on a Saturday morning with the old machine still running beside it, instead of halfway through a cutover you cannot reverse.

The cheaper question comes first: can anyone on that team make a change, reverse it, and see what happened? If the answer is no, nothing else on the roadmap is real. Finding that out is a codebase audit. Acting on it is legacy rescue. For the engineering version of how we work through a system like this one, we wrote about pointing coding agents at legacy codebases.