Takeaways from the biggest tech outage

The Issue

CrowdStrike is a provider of cybersecurity software for endpoint protection.

They rolled out an update to their software for their Windows 10 clients that breaks the installation, causing a blue screen of death.

20.000 of their business customers were impacted, some of whom have tens of thousands of devices.

Airports, stock exchanges, accounting firms, banks, hospitals, production, and air traffic controllers ground to a halt.

The cause? Human error.

I’ve lost company money before due to my stupidity, but the person who pushed that ship-it button caused mayhem on the highest level.

Interestingly, there probably was no canary deployment, or it was rolled out too fast for the bells to ring until it was too late.

The problem is compounded by a fix requiring manual intervention on each device (deleting a file).

Imagine 50,000 devices in your business suddenly going out, and you need to manually delete a file on every single one.

It’s an IT nightmare.

What can we learn from it?

Lesson 1: Eliminate the human factor

The more complex your system, the more it pays to build processes and systems that prevent a single individual from making a mistake.

Automated tests + canary deployments,

Lesson 2: Incremental deployments

Roll out a feature to 2%, then 5%, then 15% and so on. It’s crucial to wait for feedback and not go from 2% to 100% too fast.

While it may be argued that it makes deployments too slow (e.g., if you’re patching a critical vulnerability), it makes them safer.

If you use this approach correctly, I can’t imagine bricking 83.3% of your customers.

Lesson 3: Putting all your eggs in one basket

On the customer side, as a business, if you run all your systems on Windows 10 and have no redundancy, a single update can bring your entire system down.

It makes sense to diversify and eventually include Linux in the mix so you can still operate (even if at reduced capacity).

Bonus Lesson: Cyberthreats are on the rise

As more and more of our services go digital, the impact of cyberattacks (or just stupid mistakes) dramatically increases.

We’ve seen cyberwarfare play a crucial role in the invasion of Ukraine.

As we advance, it’s up to us as tech leaders to consider security’s role when building digital products.

Not just for the sake of protecting our users’ data.

But to keep our businesses from collapsing under the weight of a single botched update.

Stay resilient,

Taj

*P.S. I have 4 slots left for a ****FREE 1:1 private call ***to work on a specific business challenge that you have (300 EUR value). Because it’s free, you pay with your feedback and a testimonial (optional). Reply to this email, and I’ll send you the promo link to reserve your slot.