On July 18, 2024, CrowdStrike—an independent cybersecurity company—released a software update that unexpectedly impacted IT systems worldwide. Although this incident wasn’t directly caused by Microsoft, it significantly affected our ecosystem, prompting us to collaborate closely with CrowdStrike and other stakeholders to address the issue and support our customers¹.

Immediate Impact

• Scope: Approximately 8.5 million Windows devices (less than 1% of all Windows machines) were affected. While this percentage seems small, the economic and societal impact was substantial due to CrowdStrike’s widespread use by enterprises running critical services.
• Disruption: Businesses faced disruptions, and individuals experienced changes in their daily routines.

Steps Taken to Remediate

1. Communication and Collaboration:

• We maintained ongoing communication with customers, CrowdStrike, and external developers.
• Collaboration with other cloud providers (such as Google Cloud Platform and Amazon Web Services) allowed us to share awareness and inform ongoing conversations.

1. Technical Guidance and Support:

• We engaged hundreds of Microsoft engineers and experts to work directly with customers and restore services.
• Manual remediation documentation and scripts were promptly posted to assist affected users.
• The Azure Status Dashboard provided real-time updates on the incident.

1. Scalable Solutions:

• CrowdStrike helped develop a scalable solution to accelerate a fix for their faulty update within Microsoft’s Azure infrastructure.
• Collaboration with AWS and GCP further enhanced our approach.

Lessons Learned

1. Interconnected Ecosystem: This incident highlights how interconnected our tech ecosystem is—cloud providers, software platforms, security vendors, and customers all play critical roles.
2. Prioritizing Safe Deployment: We must prioritize safe deployment and disaster recovery mechanisms to minimize such disruptions.

In summary, while software updates can occasionally cause disturbances, incidents like the CrowdStrike outage are infrequent. Let’s continue learning, collaborating, and moving forward together!

For more technical details, you can also refer to CrowdStrike’s official blog post on the matter⁴. And remember, even in the world of technology, we’re all in this together!