A global IT outage caused by a defective software update from the cybersecurity firm CrowdStrike crashed an estimated 8.5 million devices using Windows operating systems this summer, according to a July 20 blog post by Microsoft Vice President of Enterprise and OS Security David Weston.
The incident, which may rank among the worst IT outages in history, crippled computer booking systems used by Delta and other airlines, affecting U.S. air travel for days afterward. Healthcare, banking, IT and retail/wholesale businesses were among the largest sectors hit, according to data intelligence firm Parametrix, which estimated it had a $5.4 billion impact on Fortune 500 companies.
Companies in the facilities management sector that Facilities Dive spoke with reported minimal or no impacts, however.
Cushman & Wakefield experienced issues on fewer than 10% of its Windows systems using CrowdStrike, said Eric Hart, the company’s chief information officer. “We identified the issue and solution within hours and mobilized hundreds of people to implement the manual fix.”
It took 15 to 20 minutes to recover each impacted computer, and there was no data lost as a result of the incident, Hart said.
“Our facilities operations remained unaffected and client services continued as usual thanks to our business continuity plans,” Hart said. “Most customers were unaware of any impact.” Almost none of his professional contacts “across various sectors” experienced significant service issues beyond the third day, he added.
Facilitron, which offers facility scheduling and building management products for some of the United States’ largest school districts, likewise saw “minimal impact,” CEO Jeff Benjamin said.
JLL representatives declined to comment for this article, but a Microsoft case study published in January reports that JLL had transitioned away from CrowdStrike and consolidated its cybersecurity efforts on the Tanium platform sometime before the outage occurred.
Limited impacts to their operations notwithstanding, both Benjamin and Hart advised facilities management companies to prepare for IT outages or cyberattacks with strategies such as the following:
- Create redundancies and operations disbursed across multiple tech providers. Hart and Benjamin credited their companies’ redundant approaches to digital infrastructure for minimizing the CrowdStrike outage’s impact on their operations. According to Benjamin, Facilitron has backup data centers across different cloud providers with automated system failover processes.
Cushman & Wakefield uses “a variety of third-party applications and support systems [for many clients], which has helped us avoid any service disruptions or issues,” Hart said.
- Audit third-party providers for cybersecurity. Facilitron regularly refines its vendor management processes and conducts cloud infrastructure audits to identify security threats and gaps, Benjamin said. “While third-party integrations can drive efficiency and innovation, they also introduce potential vulnerabilities,” he said. Most software products and services have integration capabilities today, he noted.
- Develop business continuity and disaster recovery plans to mitigate the fallout from IT outages. “Ultimately, [the CrowdStrike] outage was similar to any tech outage, highlighting the importance of a continuity plan,” Hart said. This is especially important for systems that rely on third-party app integrations, both executives said.
Effective IT incident responses involve experts sourced from across the affected organization and outside it as well, including management, technical leads, communications staff, and legal support and security experts, according to Internet Security, a 2003 book on digital security practices by Juanita Ellis and Tim Speed.
IT incident response plans should be designed to quickly loop in these stakeholders, especially those outside the organization, Hart said. Something as simple as keeping accurate contact information at the ready can make a difference, he added.
“When external help is needed … you can't be searching for contact information,” Hart said.
- Test those plans with exercises that simulate an attack. The U.S. Cybersecurity & Infrastructure Security Agency offers free cybersecurity “tabletop exercise” packages for dozens of industries and types of infrastructure, including commercial facilities, Hart said.
After incidents like the CrowdStrike outage, structured reviews or “post-mortems” are warranted to get a clearer picture of what happened and draw lessons for potential future incidents, Benjamin added.
