123-reg has provided more information on the error that saw some of its customers' websites "effectively deleted".
The company, which hosts 1.7 million sites in the UK, said on Monday that an error during maintenance "effectively deleted" a number of websites belonging to its virtual private server (VPS) customers.
123-reg tried to play the scale of the incident in a statement sent to V3, claiming that a small "minority" of its customers were affected.
"We suffered a technical fault which has affected a minority of 123-reg customers. The fault was limited to 67 servers out of 115,000 (across Europe) and it is important to note that only a selection of customers on the 67 servers affected has been impacted," the firm said.
"We are investigating the restoration of each VPS on a case-by-case basis and are working individually with customers to keep them informed of the website recovery process. We are working as quickly as we can to restore service to normal."
Naturally, 123-reg is sorry about the glitch and apologised to those whose websites are still missing.
"We would like to extend our apologies to affected customers and assure them that we are doing everything we can to restore their data as quickly as possible," the statement said.
Whether this appeases customers remains to be seen, with one saying after the incident the error could have caused the company to miss out on a decent whack of potential funding, as those visiting the website saw the message: '550 This domain is not hosted here.'
"This is one of six occasions over the past 12 months where the VPS service is being affected at 123-reg," the customer said.
"All of the issues have resulted from process or human error. It is clear that 123-reg does not employ sufficient expertise or maintain sufficient hardware capacity to ensure that potentially catastrophic failures do not occur.
"It is also clear that they do not have processes in place which allow them to learn from previous mistakes."
123-reg admitted the problem in an email sent to affected customers on Saturday, warning that some may have lost data if they did not have a personal backup.
"As part of a clean-up process on the 123-reg VPS platform, a script was run at 7am on 16.04.16. This script is run to show us the number of machines active against the master database. An error on the script showed a 'zero-records' response from the database for some live VPS," the message said.
"For those customers, this created a 'failure' scenario - showing no VMs and effectively deleting what was on the host. As a result of our team's investigations, we can conclude that the issues have resulted in some data loss for some customers. Our teams have been and continue to work to restore."
123-reg is working on a fix, and has apologised to those affected: "We have recovery running on the VPS servers and some are restoring to new disks. We have also begun copying recovered VPS images to new hosts and we expect some VPS to be back up and running throughout the night and into tomorrow.
"I understand that some customers may have lost some confidence in the service that we offer. I want to explain what we have done to prevent this happening again.
"We have started an audit on all cron-jobs and scripts controlling the platform, and associated architecture, so that no script will have the ability to delete images, only suspend. For image deletion for those suspended over 28 days we will have a human eye to double check.
"A new platform will be available by the end of the year which will provide self-managed and automated snapshot backups, in addition to architecture technology to back up the whole platform, something that is not available on the current platform. I hope this goes some way to win back your confidence."
V3 is hosting a Cloud and Infrastructure Summit from 20-21 April, offering an in-depth look at how cloud computing is changing the world.
Dr Kuan Hon criticises GDPR consent emails that will only eviscerate marketing databases and 'media misinformation'
Apple squashes Steam Link app on 'business conflicts' grounds
Philip Hammond wants to forget rules that the UK agreed with the EU to ban non-European companies from the satellites
Instapaper to 'go dark' in Europe until it can work out GDPR compliance