Google has published a post-mortem of an incident in February in which Google App Engine went down for over two hours.
All Google App Engine applications were "degraded" from 7:48am to 10:09am PST on 24 February after a power failure at the company's main datacentre, the firm said.
About 25 per cent of the servers failed within five minutes owing to a delay in back-up power generation. Google's message boards started showing questions from users almost immediately.
"By this time, our primary on-call engineer had determined that App Engine is down," the report said.
"The on-call engineer, according to procedure, paged our product managers and engineering leads to handle communicating the outage to users. A few minutes later, the first post from the App Engine team about this outage is made on the external group."
There was confusion about the instructions for switching to a back-up datacentre and the decision-maker for the crossover could not be found. The team then received data suggesting that the datacentre was recovering and that a changeover was not neccesary.
However, the data turned out to be inaccurate and this extended the outage considerably. By the time the move to the backup servers had been made, Google's App Engine had been down for more than two hours.
The report found that Google had not developed plans for a partial datacentre failure, nor for determining whether the datacentre was able to continue running on such a reduced server count.
The company will now hold regular drills for failure, with a wider spectrum of possible situations, and a bi-monthly audit of all operations documents.
Google claimed that a similar failure today would cause a service slowdown for a maximum of 20 minutes with the new procedures, rather than a complete outage.
Latest Tesla news: Tesla share price continues to fall after Saudi Arabia's sovereign wealth fund is linked to investment in rival
SEC 'probe' takes its toll on Tesla as new research suggests that Tesla loses $6,000 on every $35,000 Model 3
RTX 280 Ti will come with 11GB of fast GDDR6 video RAM with a 352-bit memory bus offering 616Gbps
The scale of jobs lost to automation will be at least as large as those in the first three industrial revolutions
10nm Cannon Lake Core i3-8121U CPUs make a rare outing with Intel's NUC mini PC