Story: Losing production data twice

Let me tell you a horror story about losing production data twice on the same day. Yes, you heard me right. Twice. Because once is not enough for me.

I was working on a new feature for a client's application, and I needed the most recent production data to verify if the functionality worked.

I logged in on the production machine, made a "backup", copied it to my device, and "restored" it on my machine. But when performing the final checks, I noticed that the most recent entry in the database looked like the tests I had done earlier today. I double-checked, and it was test data from that morning. I didn't perform the restore. I was tired at that point, so I tried again. After restoring the backup, I noticed my test data was still in the database.

At this point, I started to panic; I logged back into production, copied the file to my machine, and restored it. The data was different this time, but the last entry was more than two months ago.

I felt a surge of panic as I realized what I had done. I had "restored" the two-month-old backup on production. And on my local machine, I had "backed up" instead of restoring my database. I was convinced that the customer had lost two months of data. And this was a financial database of a company that performed transactions involving millions of data. Due to my actions, their accounting department lost two months of financial transactions.

Seeing no way of restoring it, I confessed to my coworker what had happened. But for some reason, he smiled when I told my story. I was convinced we were screwed. After telling him the whole story, he told me we could still recover two months of data.

Early that morning, my coworker was researching a production issue and had also made a backup. The file was still on his USB stick since his computer didn't have enough space. He reached into his bag and gave me the stick. As I plugged his USB stick into my machine, I thanked him from the bottom of my heart. I connected to the production database and clicked the buttons to restore his backup. Since restoration should only take a few minutes, I was sure the mistake would soon be in the past until I noticed that the screen said: "Writing backup to file... 1%"

In my attempt to quickly restore the problem, I performed the exact steps again, and what was worse: I hadn't copied the file to my drive first. The file that was being overwritten was the last remaining backup. For the second time!

I looked in horror at my coworker. He asked me what I did. I explained that instead of restoring, I had backed up again. His eyes when wide, first from the shock but then from laughing.

As luck would have it, he had restored the production backup on his development machine just minutes before handing me the USB. So his local development database was still untouched. We backed up his database and then restored that backup to production. This time we double-checked every step.

That day, I learned three valuable lessons: always back up your data twice and always have a coworker who has your back. And finally, never give a developer access to production.