Our colocation facility bounced the power again (apparently in order to deal with the failed UPS that caused our previous incident.) We are on route to the facility to bring servers back online now and that should be done shortly. We have been and will continue to work with Yahoo! to migrate to better facilities as fast as humanly possible.
Update (10:50 PM EST): We are at the facility and are bringing up machines now.
Update (11:14 PM EST): Bringing up slaves now. Should be done shortly.
Update (12:09 PM EST): All but one server up now. Bringing up inbox, search, etc.
Update (12:20 PM EST): The facility will be cutting power back over to the UPS at 1 AM. Supposedly this will not affect service, but neither was the test that brought everything down... cross your fingers.
Update (2:04 AM EST): Still waiting for the cutover back to the UPS. They're planning to give us an hour warning, and have not done so yet. We'll bring down the site beforehand so we can boot cleanly in case they drop power again.
Update (6:37 AM EST): They just cut over power to the UPS and bounced power again. We're bringing stuff back online again, everything should be good in a few minutes.
Update (6:57 AM EST): Everything is back online.
More from joshua in
Yahoo News