In a rare admission of fault, Google has acknowledged that a configuration error on their part was responsible for a recent outage experienced by Australian superannuation giant UniSuper and their 630,000 customers. In effect, the fund’s private cloud was inadvertently deleted. The incident, which occurred on April 18th, 2024, left UniSuper members unable to access their online accounts for over seven days.
The industry fund manager holds more than AUD$130 billion in its accounts. The Australian Prudential Regulation Authority (APRA), Australian Prudential Regulation Authority, is reported to have said it was “aware of the system outage affecting services and is continuing to monitor the situation”. It is unclear if APRA will take any action in response to this incident.
Disaster Recovery Remains Critical In The Cloud
According to a joint statement from Google, a “misconfiguration issue” within its cloud platform caused the disruption. While Google hasn’t elaborated on the specifics of the misconfiguration, the incident highlights the potential pitfalls of relying on third-party cloud services. UniSuper in the joint statement claims they had duplication in two geographies as a protection against outages and loss.
However, when Google deleted UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies, in effect making any form of business continuity or disaster recovery impossible beyond a total rebuild and subsequent restoration of data from backups, as UniSuper reports they had backups occurring with another yet to be named cloud service provider.
Customers Are Asking For Answers Beyond An FAQ
In a Frequently Asked Questions (FAQ) style bullet point list of “pull down” items which require a manual click of the mouse to make visible, UniSuper has provided a very high level “outage update”, where buried under a pull down clickable list item, Google Cloud CEO, Thomas Kurian is quoted as having confirmed that “the disruption arose from “an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription”, and that “while supporting UniSuper to bring its systems back online, Google Cloud has been conducting a root cause analysis”.
In a statement made to the Financial Standard publication, UniSuper is reported to have said “Member data is safe – no UniSuper data has been exposed to unauthorised third parties as a result of this incident”, the statement goes on to say “We have encountered absolutely minimal data loss during our progressive restoration of services, with this data being predominately related to our internal operating processes. Members can be assured all member data is rigorously backed up and these backups are secure. UniSuper does not expect this to have any impact on members”. It is unlikely the 630,000 effected UniSuper customer will consider any of this as a reasonable response or reassurance, and understandably now want answers from the industry super fund manager about whether their balances are safe and how a week-long outage happened.
Business Impacts of the Outage
For UniSuper, and it’s Cloud Hosting service provider, American hyper-scaler cloud services company Google, the outage resulted in several negative consequences:
- Disrupted Customer Service: The inability to access online accounts meant UniSuper members were unable to view their superannuation balances, make contributions, or update their personal details. This disruption could have caused frustration and inconvenience for members, especially those nearing retirement or planning significant financial decisions.
- Operational Delays: UniSuper’s internal operations were also likely impacted by the outage. Staff may have been unable to access essential member data or process transactions during the downtime, potentially causing delays in customer service response times.
- Reputational Damage: Any service outage can damage a company’s reputation, especially for financial institutions like UniSuper where trust is paramount. The outage might have shaken some members’ confidence in UniSuper’s ability to securely manage their financial data.
Customer Impacts Beyond Account Access
While regaining access to accounts was undoubtedly the primary concern for UniSuper members, the outage had broader implications:
- Heightened Security Concerns: News of an outage caused by a cloud misconfiguration raises concerns about the security of member data stored on those platforms. UniSuper will need to assure its members that their data remains secure and that steps are being taken to prevent similar incidents in the future.
- Limited Communication Options: During the outage, members who typically rely on online channels to contact UniSuper may have faced challenges reaching customer service representatives. This lack of communication could have further amplified frustration for members already dealing with inaccessible accounts.
- Disruption to Financial Planning: UniSuper members may have had financial plans or decisions on hold due to the outage. Being unable to access account information or make contributions could have caused delays or forced them to postpone important financial decisions.
Google Cloud Outage Raises Questions About Reliability
The Google Cloud outage affecting UniSuper could have a negative impact on Google’s company brand in a many ways, key among which would be:
- Damaged Reputation for Reliability: This incident highlights a potential weakness in Google Cloud’s infrastructure or configuration management. Customers may question the platform’s reliability, especially those considering cloud migration.
- Reduced Trust in Cloud Security: News of the outage raises concerns about data security on Google Cloud. Businesses entrusting sensitive data to the platform may be less confident after this incident.
- Potential Loss of Business: If Google Cloud is perceived as less reliable or secure, it could lose potential customers to competitors offering cloud services.
By taking responsibility for the outage, explaining the cause, and demonstrating efforts to prevent future occurrences, Google can mitigate the reputational damage. However, regaining full trust may require a period of improved performance and a continued focus on robust security measures. Other businesses using Google cloud computing services will no doubt not be paying attention and considering what the impact to their organisations and customers might be if this type of mistake were to happen to their cloud hosted environments, be it with Google, or similar cloud hosting service providers.
The Importance of Cloud Service Reliability
The Google Cloud outage affecting UniSuper yet again underscores the critical need for cloud service providers to maintain robust infrastructure and rigorous configuration protocols. Businesses migrating to the cloud must carefully evaluate the reliability and redundancy measures offered by potential providers.
While outages can occur even with the best practices in place, transparency and clear communication are crucial during such events. Google’s acknowledgement of the issue is a positive step, but a more detailed explanation of the misconfiguration and the steps taken to prevent future occurrences would further reassure UniSuper and its members.
The incident also serves as a reminder for businesses to have robust contingency plans in place for cloud-based outages, in the form of both a Disaster Recovery plan (DR) and Business Continuity Plan (BCP) and to have the capability to enact either or both immediately in such circumstances as a service provider deleting their entire business systems with the click of a button, either by accident or otherwise. Redundant systems and alternative communication channels can help minimise disruption and maintain a level of service during unforeseen circumstances.
As cloud adoption continues to rise, ensuring the reliability and security of these platforms remains a top priority for both cloud providers and businesses that leverage their services. For organisations where Information Technology is not their core business, incidents such as this should act as an immediate call to action to review and possibly renew or update their 3rd party partnerships, suppliers and ensure sufficient support is in place such that they have business partners under contract to support them in the case of partial or entire outages or loss of business systems.
