Google has blamed a software bug for an outage on its web services including Gmail, Google+, Calendar and Docs which left them inaccessible on Friday.
Writing on the Google blog, the firm's vice president of engineering Ben Treynor explained that the problems, which affected most users, lasted for around 25 minutes, while some were unable to access services for almost an hour.
The outage began at 10.55am PST (6.55pm GMT) following a software bug, which caused an internal system to send an incorrect configuration to services for 15 minutes, Treynor said. As a result, when users attempted to access Google services, the requests were ignored and they received error messages.
Treynor wrote: "Users began seeing these errors on affected services at 11.02am, and at that time our internal monitoring alerted Google's Site Reliability Team. Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at 11.14am and began sending it; errors subsided rapidly starting at this time. By 11.30am the correct configuration was live everywhere and almost all users' service was restored.
Treynor apologised for the outage, admitting Google had "missed the mark" on its targets for reliability. He said Google's engineering team was now focused on eliminating the bug that caused the initial problem and that it was also working to speed up its response for the future.
He said the team would be adding additional validation checks for configurations, so incorrect ones would no longer be sent, as well as "more targeted monitoring" to more quickly diagnose problems.
Last year, the firm admitted that a glitch causing a Gmail outage was caused by a "rare" double network failure, highlighting the sometimes fragile nature of services used by a large number of consumers.
In fear of future shortage - or in preparation for its own electric car project?
But if you're running anything older you'll have to wait
Powered by servers based on Qualcomm's scalable 48-core Centriq 2400 10nm CPUs
Malware has been in circulation for more than a year