In case you didn’t know CrowdStrike makes anti-malware software for enterprises and they pushed out an update late last night that basically broke Windows machines running it. These machines need to be manually fixed via a process that involves either booting Windows into recovery or safe mode and then manually deleting the offending driver file. If your drive is encrypted this is made far more difficult by you having to manually key in your Bitlocker recovery key, assuming you even have access to it at all. This is a nightmare for end users as well as IT people.

Fuck CrowdStrike

The handful of those reading this blog already know this: I’m tried of watching the tech industry build things that are prone to failure. It pisses me off. It should piss everybody off and getting the word out is one of the reasons I started this blog. This blog is anonymous because I can’t speak freely and expect to keep my career from spontaneously combusting. That having been said, at my company all of our environments went down because all of our Windows machines are running this shitty software. Prod is just now getting back online after IT busting its ass for six plus hours to fix hundreds of servers. A lot of our Windows laptops (mine included) also went down. I managed to fix mine because I was one of the lucky ones whose Bitlocker recovery key was stored in Azure like it was supposed to be.

It’s not just my company. Airlines, Hospitals and 911 call centers are down. People’s lives are on the fucking line and big tech has absolutely completely failed them today. People will die because CrowdStrike fucked up. People will die because the orgs that use Crowdstrike didn’t see a problem with using the exact same thing that everybody else uses.

This is a fucking disaster. It was also totally avoidable. Will the industry learn it’s lesson? Probably not.

There are so many terrible trends in this industry that are part of the problem here. The short list is:

  1. Windows: For the love of God. If your purchased software relies on it, get new software. If your custom software relies on it, port it. Windows is a dying platform and this incident will only serve to hasten it’s demise.

  2. Anti-Virus / Malware: It’s long past time we figured out that software like this is less useful than tits on a boar. It’s always a day late and a dollar short as this shit is basically defective by design. In addition, the amount of trust you have to give this software is generally rewarded with shittier performance and decreased stability.

  3. Copy Cats: Most orgs are using a lot of the same shit as everybody else. It doesn’t matter what product we are talking about. It could be software like Adobe CS, CrowdStrike, Windows, Microsoft Office or whatever else you can think of. It could also be cloud services like AWS, Azure and Google Cloud. This industry is infected with the delusion that doing what everybody else does is the safest possible option. We are building fallible things that depend on other unaccountable entities to function and because we are largely using the same things, we are setting ourselves and our customers up for EXACTLY THIS KIND OF FAILURE.

This can only end poorly. As for CrowdStrike itself, the company should go straight to zero. Sadly their stock is only down 10% at the time of this writing. That in and of itself is a travesty. If I was holding CRWD right now I would’ve already taken whatever action I needed to immediately divest myself of it.

This might be the biggest IT outage of all time and it’s their fault. If that company’s stock doesn’t go to zero, victims (yes we are all victims of CrowdStrike’s malfeasance) don’t attempt to sue them out of existence and orgs don’t immediately decide to terminate their contracts with Crowdstrike and remove their software then that basically means that the feedback mechanism in this industry is completely broken. This shouldn’t come as a surprise to anybody who is regularly reading this blog because you all already know the score.

But for anybody who has shown up here for the first time ever, let me say it loud and clear: The tech industry is broken and it is breaking the back of the world. It lies and cheats its customers. It wastes our limited natural resources. It creates systems that are inherently fallible and attempts to just brush off the inevitable failures that result from employing defective design practices.

Perhaps the real takeaway is that instead of increasing complexity to hedge against the ever increasing level of complexity in tech, we should instead focus on building less complex mechanisms with fewer interdependencies that can survive crisis level incidents such as this. That was the original point of the internet (aka ARPANet) after all. To provide the United States Department of Defense with a decentralized network that could remain functional in the event of a nuclear attack taking out other parts of that network.

Finally to the IT people out there working feverishly to get shit back online: You got my sympathy. Cause this shit sucks.