From the FinallyFast Help Desk:
On Monday, Ben Treynor, VP Engineering and Site Reliability Czar (24×7) posted on Google’s Blog that an update to e-mail storage software led to an unexpected bug Sunday that left tens of thousands of Gmail users without access to their messages:
“Imagine the sinking feeling of logging in to your Gmail account and finding it empty. That’s what happened to 0.02% of Gmail users yesterday, and we’re very sorry. The good news is that email was never lost and we’ve restored access for many of those affected. Though it may take longer than we originally expected, we’re making good progress and things should be back to normal for everyone soon.”
Treynor goes on to expose the culprit:
“So what caused this problem? We released a storage software update that introduced the unexpected bug, which caused 0.02% of Gmail users to temporarily lose access to their email. When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version.”
Luckily, the emails in question were not lost forever:
“I know what some of you are thinking: how could this happen if we have multiple copies of your data, in multiple data centers? Well, in some rare instances software bugs can affect several copies of the data. That’s what happened here. Some copies of mail were deleted, and we’ve been hard at work over the last 30 hours getting it back for the people affected by this issue. To protect your information from these unusual bugs, we also back it up to tape. Since the tapes are offline, they’re protected from such software bugs. But restoring data from them also takes longer than transferring your requests to another data center, which is why it’s taken us hours to get the email back instead of milliseconds.”
Treynor went on to say that a detailed report outlining what happened would be posted to Google’s Apps Status Dashboard, with notes on corrective actions they would take to avoid this happening in the future. He also pointed out:
“it’s important to note that email sent to you between 6:00 PM PST on February 27 and 2:00 PM PST on February 28 was likely not delivered to your mailbox, and the senders would have received a notification that their messages weren’t delivered.”
The post was later followed up with an update on Tuesday explaining that data for the remaining 0.012% of affected users had been successfully restored from tapes and is now being processed. Google would eventually migrate the data back plan to into mailboxes, and in the hours that follow users will eventually regain access to their data. Accounts with large amounts of mail will probably require a longer recovery time.
