March 11, 2020, 00:47 a.m. (Paris time): A fire breaks out in the OVH Cloud data center in Strasbourg. Almost immediately, the entire data center is powered down.
As a reminder: this data center hosts the Auditool application as well as all the audit data of some of our customers.
March 11, 2020, 3:50 a.m.: The automatic daily backup of Auditool data to a second site in France fails, due to the unavailability of the data center.
March 11, 2020, 7:00 a.m.: The fire is brought under control thanks to the intervention of a hundred firemen. A part of the data center was saved but the power is cut for the whole site and the whole data center is still out of service.
March 11, 2020, 8:00 a.m.: The JUNE crisis management team is activated. We take stock of the impact and immediately inform our customers of the incident. 30% of our customers are impacted.
We quickly understand that the data saved in the OVH datacenter will, at best, be recovered in several weeks, at worst, lost. Fortunately, each Auditool license includes a daily external backup.
March 11, 2020, 10:00 a.m.: The Auditool recovery plan is activated, but the OVH domain name management service is unavailable due to excessive traffic. Our teams are forced to wait for the restoration of the OVH service to continue the recovery plan.
March 11, 2020 2:00 p.m.: OVH's DNS service is operational again. Our teams replicate and deploy Auditool on a new server. Operational tests are also performed to ensure proper deployment.
March 11, 2020, 4:30 p.m.: Full recovery of the Auditool service, 100% of our customers have access to their Auditool instance, with the data from the previous day's backup. Auditool is now based on the OVH site in Gravelines, France. The application is hosted on a server in France, with the same performance and data protection policy.
Thus, Auditool will continue to offer the same quality of service as before, and all our data protection commitments remain valid.
Small hitch: the data for the day of March 10 was lost... But not for Auditool! Thanks to Auditool's offline mode, our customers were able to continue working on their audit even during the downtime and synchronize the two days of work at the end of the day on March 11.
The result: no audit data lost and almost no impact on our customers' productivity.
A hectic day for our teams, fortunately without consequences thanks to outsourced backups and a well established recovery plan.
What's next? We are waiting to hear from OVH about data recovery and the resumption of service. This incident also allowed us to test our recovery plan in real conditions, which we will continue to improve.
The June Factory team apologizes for the inconvenience caused by this exceptional incident beyond our control.
We remain at your disposal for any question or complaint.
Photo Credit : Twitter
