Author Archive

Blogger was unavailable for most users for over an hour. Service has now been restored.

RESOLVED: The issue affecting Blogger users in Europe has been resolved. There was a configuration problem in a new datacenter. We will provide a postmortem on Monday. Thanks to all who assisted us analyzing the problem.

Postmortem on recent issues affecting users in Europe

What happened
Beginning at approximately 2:30am (PDT) Saturday, October 31, a DNS configuration issue in a new datacenter in Europe caused requests to blogger.com and *.blogspot.com to be sent to an invalid IP address. Affected users throughout Europe were unable to connect to either Blogger or to blogs hosted by Blogger at blogspot.com for approximately 10 hours.

What we're doing about it
1. Blogger's systems engineers will proactively monitor and verify DNS settings during major system changes.
2. We have improved our automated monitoring so that on-call engineers will be paged earlier for DNS-related failures.
3. Checks in our DNS management system have been made to prevent the change that caused this to occur.

Eddie Kessler, Blogger Engineering Manager

By Eddie Kessler, Blogger Engineering Manager

After a recent service interruption, we started talking about how we could improve our communications about these (hopefully infrequent) issues. Going forward, in the case of significant service interruptions, we plan to publish a post mortem on the Blogger Status blog within 3 business days to provide details about what went wrong and what we're doing to help prevent similar problems in the future.

During any outage, we try to keep the Blogger Status blog updated once we know about an issue. Of course sometimes — as was the case on Saturday — affected users who cannot reach Blogger cannot reach the Status blog either. In such cases, we will try to post updates on the Blogger account on Twitter to keep users apprised of what we know and when we expect a resolution.

We don't like it when our users experience problems like what we saw on Saturday, but we hope the combination of transparency around these issues and our commitment to learn from our mistakes will help assure you that we're doing everything we can to keep Blogger a robust and reliable service for you. As always, thank you for using Blogger.

Post Mortem for Blogger Service Interruption June 7

By Eddie Kessler, Blogger Engineering Manager

As we recently committed to you, when serious service interruptions occur, we will provide you details about what went wrong and what we're doing to help prevent similar problems in the future.

From about 12:30 AM PST on Monday, June 7 until about 12:30 PM, a significant subset of Blogger users, mostly in the US Midwest, were unable to post to their blogs. No data was lost and blogspot was largely unaffected, so your blog’s readers weren’t impacted.

We posted the issue on the Blogger Status blog, and we made a brief status update on our Twitter account as well. Meanwhile, we kept a close eye on the reports coming through our help forum and Twitter stream.

The problem was caused by human error. One of our datacenters inadvertently had the wrong server configuration when we performed our regular service update. Once we figured this out, we corrected the problem quickly.

We are in the process of adding additional monitoring that will prevent this kind of outage from recurring.

We are truly sorry for any inconveniences this might have caused you. In the meantime, we will try harder to make Blogger a more reliable service.