Cybersecurity vendors often emphasize the catastrophic potential of cyber threats and offer an array of products and services to shield us from such dangers. However, a recent incident involving a faulty update from one of these vendors highlighted just how vulnerable our systems truly are. This incident not only caused significant disruptions across multiple sectors but also served as a stark reminder of the risks inherent in the very solutions designed to protect us. Bill Mew delves into what transpired, the impact, and the crucial lessons learned.
The Impact
Travelers eager to catch flights on the morning of Friday 19 July found themselves facing massive delays as the Federal Aviation Administration (FAA) in the United States grounded major airlines such as United, Delta, and American Airlines. This disruption was not confined to the US; it was a global issue, with over 1,000 flights cancelled worldwide. Passengers of Ryanair in the UK, as well as those in India, Australia, and other regions, experienced similar chaos.
The disruption extended beyond air travel. Various sectors were affected, including healthcare, where the NHS in the UK reported that while most GP practices were impacted, while thankfully emergency services remained unaffected. Other sectors also felt the strain, with train operators experiencing delays, retailer having trouble processing payments and organisations of all kinds struggling to provide services as usual.
The Problem
Initial fears of a widespread cyber attack were quickly dispelled when it was revealed that the issue originated from a Windows update issued by cybersecurity firm CrowdStrike. Although CrowdStrike promptly issued an apology and assured that the problem had been “identified, isolated, and a fix deployed,” the reality was far more complex. The faulty update caused what technicians refer to as the “blue screen of death (BSOD),” crashing computers and necessitating manual restarts in ‘Safe Mode’ to remove the faulty update and install the fixed version. This process is labor-intensive, particularly on a large scale.
This incident, described by Elon Musk as the “biggest IT fail ever,” demonstrated that even cybersecurity vendors could inadvertently cause significant disruptions. As the magnitude of the incident became apparent, CrowdStrike’s shares plummeted by 20% in unofficial US trading, translating to a $16 billion loss in value.
The Lessons
As services gradually return to normal and IT staff work tirelessly to restore systems, regulators are left questioning how such a failure could occur. The evolving cyber threat landscape has made obtaining cyber insurance increasingly challenging, with premiums rising and policy conditions becoming more stringent. Clients must now demonstrate robust cybersecurity measures, including both detection and prevention capabilities, and comprehensive staff training to ensure good cyber hygiene. Additionally, fire drills are required to test the response of senior management and critical functions to potential cyber incidents.
Insurance policies also impose strict requirements for the timely implementation of patches and security updates. Ironically, it was the automatic installation of such an update that led to this massive disruption. The failure of CrowdStrike to verify the integrity of their software update before its release is a glaring oversight.
Critcal Impact
Many organisations found themselves falling back to manual processes – with doctors surgeries, many of which had stopped taking calls as they sought to push all requests online, having to pick up the phones again. At Heathrow airport security checks slowed considerably as barcodes on boarding passes could not be read and staff needed to check everything manually.
A major aspect of crisis management is having a back up plan and knowing what processes can be done manually in the event of an outage. The plan also needs to be rehearsed so that senior management and those in critical functions know what to do in a crisis.
Immersive cyber incident simulations are the best way to do this. While such fire drills are essential for all critical infrastructure, they are also mandated under GDPR for almost all other organisations – as it is a regulatory requirement under GDPR to test and assess all systems and processes.
The Baltic Hub in the northern city of Gdansk, Poland’s largest container terminal – handling up to 2.9 million containers annually, was a major infrastructure operator to be affected by the recent incident. On Friday it asked companies to stop sending containers to the port, as it said: “Please be advised that we are struggling at the Baltic Hub with a global Microsoft operating system outage that is hampering terminal operations.”
“The potential economic impact of such incidents is immense,” suggested Simon Osborne from MSS Alliance CIC (the Maritime Safety and Security Alliance) which has recently teamed up with IARCC.org (The International Association of Risk and Crisis Communication) to develop immersive cyber incident simulations for the maritime sector. A recent cyber incident at ports in Australia brought imports and exports to a standstill for days, causing massive systemic impact.
These incidents show how vulnerable all organisations are to supply chain threats – the impact to your organization if it is compromised by the actions of a trusted supplier (as happened with Crowdstrike) or if a critical supplier that you rely on is itself taken out – such as banks, ports and power companies – as happened in Australia.
While there will be a lot of red faces at Crowdstrike as it faces regulatory scrutiny right now, if the next cyber incident takes out your operations or the operations of a critical service that you rely on and it turns out that you didn’t carry out the necessary preparation and fire drills then you’ll be the one facing the music.
This incident underscores the need for thorough vetting and testing of cybersecurity updates to prevent similar crises in the future. It also highlights the delicate balance between maintaining robust security measures and ensuring that these measures do not themselves become sources of vulnerability. The lessons learned from this event must drive improvements in how cybersecurity solutions are developed, tested, and deployed, ensuring that the tools designed to protect us do not become the catalysts for widespread disruption.
