Science Microsoft/CloudStrike Outage

[Mark]

Widespread technology outage disrupts flights, banks, media outlets and companies around the world

Microsoft appeared to suggest in its X posts that the situation was improving but escalating outages were still being reported around the world hours later.

apnews.com

I’m not entirely sure about all of the details, but apparently it has contributed to issues with everything from your average workday applications to international flights and banking. Three of the repossession agencies I do work with have been crippled, and they’re not nearly as dependent on a constant connection as some other systems.

Have any of y’all been affected?

 

[Fool's Requiem]

I don't believe this Cloud Strike issue has hit us. We probably don't even use it.

 

[Mark]

I don't believe this Cloud Strike issue has hit us. We probably don't even use it.

I just got a message from my girlfriend saying that their phones are offline at her work, and she does call center work for utility companies. Another friend has zero connectivity to his job since he works remotely, all he can do is what can be saved and submitted later on when service is fully restored.

I’m not tech savvy enough to know exactly what went wrong beyond that it was a patch update that rolled out and caused the issue.

 

[dimmerwit]

I’m not tech savvy enough to know exactly what went wrong beyond that it was a patch update that rolled out and caused the issue.

The real issue seems to be that affected systems can't be remotely fixed, which is why the impact will be needlessly long.

 

[Fool's Requiem]

Every computer connected that received whatever security update that was released by Cloud Strike is essentially borked. It's making it so that people can't even log in because their computer gets the blue screen of death when Windows is starting up. As a result, unless the computers effected have a restore point that they can recover to, they all will likely need to be reimaged. If those computers are remote workstations, it'll take longer for them because they'll need to take their computers to the office and some people are really far away from where their office is (or they REFUSE to drive into the office).

If Windows won't even boot up, there is no way to revert the update. They might be able to get in Win 11 ISO onto a thumbdrive if they have another computer that is unaffected and recover the image that way. But most people are not tech savvy enough to know how to do that, even though ever since Windows versions became free updates, it became INSANELY easy to reimage your computer without losing any data.

I would certainly hope that Airlines and banks have restore points for all of their computers, but if this is taking as long as it is, I'm guessing that's not the case.

 

[Disgustipated]

Every computer connected that received whatever security update that was released by Cloud Strike is essentially borked. It's making it so that people can't even log in because their computer gets the blue screen of death when Windows is starting up. As a result, unless the computers effected have a restore point that they can recover to, they all will likely need to be reimaged. If those computers are remote workstations, it'll take longer for them because they'll need to take their computers to the office and some people are really far away from where their office is (or they REFUSE to drive into the office).

If Windows won't even boot up, there is no way to revert the update. They might be able to get in Win 11 ISO onto a thumbdrive if they have another computer that is unaffected and recover the image that way. But most people are not tech savvy enough to know how to do that, even though ever since Windows versions became free updates, it became INSANELY easy to reimage your computer without losing any data.

I would certainly hope that Airlines and banks have restore points for all of their computers, but if this is taking as long as it is, I'm guessing that's not the case.

This isn't quite accurate. Recovery can be accomplished by deleting a specific .sys file from within the Windows directory. If the machine can be booted in safe mode, you can delete the file, or you can pull the drive and delete it. All that of course assumes you aren't running BitLocker or if you are, you are able to access the BitLocker Recovery key. It does require physical access, but after a few BSODs Windows will prompt you to choose startup options usually, and you can get to safe mode from there. Just going to be a lot of hand holding and walking through for remote workers to get them back up and running, but for the most part should NOT require a reimage.

Mercifully the MSP I own uses SentinelOne so we are not involved with this clusterfuck lol

 

[Ben]

This is widespread bad. As far as I understand it pushed a windows update that was bad and has servers and laptops stuck in the reboot loops and such.

I'm thinking the chances of it coming from causing some attack are high. This is comimg from seeing a ton of fraudulent merchant testing across the MasterCard network this morning.

 

[Disgustipated]

This is widespread bad. As far as I understand it pushed a windows update that was bad and has servers and laptops stuck in the reboot loops and such.

I'm thinking the chances of it coming from causing some attack are high. This is comimg from seeing a ton of fraudulent merchant testing across the MasterCard network this morning.

Can't attack systems that aren't running, so CrowdStrike is definitely the best there is!

 

[canadaguy]

Me upon logging into my work pc and receiving a message "it requires a reboot to install a new update":

 

[Disgustipated]

Don't worry, you're good. The CrowdStrike update doesn't require a reboot to apply, it just blue-screens the machine as soon as it downloads the update lol

 

[Ben]

Can't attack systems that aren't running, so CrowdStrike is definitely the best there is!

View attachment 18349

The Mastercard network itself isn't reporting any outages, but lots of banks are. A lot of those banking services may still function to run the mastercard network initiation, but backend systems like processors or fraud detection systems might not be.

Bad case, your bank has nothing to block widespread BIN testing against cards on their portfolio. That's where the criminal gets the first 6 digits of a mastercard, visa, etc. which is the Bank Identification Number and represents the range of card numbers that bank that owns it can use. In a way, every possible card number in that range exists, but your bank might not have an account linked to it for it to come back as anything but 'invalid'.

Anyway, worse case the bank can't even talk to Mastercard, and when the response times out MC decision it based on stand-in rules. Which are kind of like n configuring your router, a lot might not have done much with MC rules and possibly be approving things without your bank even seeing it.


I spent my morning blocking merchants that're being used to test for open cards, including things like open AI and Uber.

 

[Ben]

Oh, also the merchants themselves might just be fucked. I have also heard of problems with gas stations like Exxon, Speedway, Pilot.

Good, if not busy, time to be an IT guy I guess.

At least it is a global thing, and our military enemies are probably just as fucked and aren't going to attack attack. I just foresee a lot of cyber attacks due to systems being down and vulnerable, financial, data, etc.

 

[shortkut]

It’s bricked half the computers here

 

[canadaguy]

Don't worry, you're good. The CrowdStrike update doesn't require a reboot to apply, it just blue-screens the machine as soon as it downloads the update lol

You say "don't worry" but I was looking at it as "if my computer doesn't work that's a free long weekend"

 

[Rachel]

I have been hearing about big problems in hospitals, they have to revert back to how they did things in the 60s and 70s with pencil and paper.

 

[Ben]

State services like the DMV and 911 are down in a lot of places. I imagine smaller towns hit worse.

CrowdStrike:

 

[Disgustipated]

I doubt smaller towns are willing to pay for CrowdStike or are even keeping any semblance of solid IT infrastructure tbh

 

[Ben]

I doubt smaller towns are willing to pay for CrowdStike or are even keeping any semblance of solid IT infrastructure tbh

Fair point... I guess below a certain financial level they're insulated from that.

 

[Ants!]

Computers here at the library I work at are working fine.

 

[The Eye in the Sky!]

It's so obvious.

 

[Fool's Requiem]

You can almost buy NCAA 2014 for the same price as the previous games thanks to that game.

 

[Gloom-is-good]

My team basically deals with online banking all day so I sent them all home early. Most of what we do with our partner banks is fine, it's s the whole other half of what we use that isn’t working. Super annoying

Although this morning when my husband told me I was like "would this be perfect to do crimes?" Just saying if student/medical/credit card debt gets erased, I wouldn't be mad

 

[dimmerwit]

I'm so glad we have the press to fully explain what happened when someone "clicked on the wrong thingummyjig wotsit".

We need to find this nerd/poor sod and hunt them down.

 

[Mark]

View attachment 18417

I'm so glad we have the press to fully explain what happened when someone "clicked on the wrong thingummyjig wotsit".

We need to find this nerd/poor sod and hunt them down.

I love the special USA section… we really are living in tabloid headlines.

 

[Friel]

It stopped our fuel cards from working, which I only found out after I filled the tank of the work van. Had to pay using my own money. Was a great start to the morning.

 

[Ben]

It stopped our fuel cards from working, which I only found out after I filled the tank of the work van. Had to pay using my own money. Was a great start to the morning.

Who do you use, out of curiosity?

 

[Friel]

Who do you use, out of curiosity?

Sorry only seeing this now. We use Diesel Card Ireland (DCI).

 

[Lunar Kreskents]

I doubt smaller towns are willing to pay for CrowdStike or are even keeping any semblance of solid IT infrastructure tbh

Fair point... I guess below a certain financial level they're insulated from that.

Coming from a very small town village, my feeling would have been more like... what are these "services" you speak of?

 

[Ben]

Yeah... It definitely wound up causing issues for me at work and led to a ton of meetings.

This is the core of Crowdstrike's explanation of what happened. Most of the stuff before that was explanation of each of these terms, but it comes down to "bad QA content validation allowed a badly coded template into production, which led to it causing a system memory problem, that led to BSOD."

Article:

What Happened on July 19, 2024?
On July 19, 2024, two additional IPC Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data.

Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production.

When received by the sensor and loaded into the Content Interpreter, problematic content in Channel File 291 resulted in an out-of-bounds memory read triggering an exception. This unexpected exception could not be gracefully handled, resulting in a Windows operating system crash (BSOD).

Source: https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/

Article:

The bad release was published just after midnight Eastern time on July 19, and rolled back an hour and a half later, at 1:27 a.m. Eastern, CrowdStrike said. But by then millions of computers had already automatically downloaded the faulty update. The issue affected only Windows devices, not Mac or Linux machines, and only those that were switched on and able to receive updates during those early morning hours.

Thanks to the timing of the incident, organizations in Europe and Asia "had more of their work day affected by the outage, unlike the Americas," Fitch wrote in its blog post.

When Windows devices using CrowdStrike's cybersecurity tools tried to access the flawed file, it caused an "out-of-bounds memory read" that "could not be gracefully handled, resulting in a Windows operating system crash," CrowdStrike said.

That's the Blue Screen of Death that many people reported seeing on their machines, and that only a manual intervention to delete the bad file could fix — a slow, painstaking process when you consider that as many as 8.5 million individual devices will need to be reset this way.

That figure is small as a percentage of the wider Windows ecosystem, said Microsoft — a company that played no direct role in the outage. Still, Microsoft said in a blog post, it "demonstrates the interconnected nature of our broad ecosystem."

Source: https://www.cnn.com/2024/07/24/tech/crowdstrike-outage-cost-cause/index.html

So that's dumb. Microsoft of course wants to take the "see we told you we needed to limit access to the kernel" route, which on one hand is fair if they foresaw something like this coming...

Article:

While CrowdStrike has blamed a bug in its testing software for its botched update, its software runs at the kernel level — the core part of an operating system that has unrestricted access to system memory and hardware. This means that if something goes wrong with CrowdStrike's app, it can take down Windows machines with a Blue Screen of Death.

CrowdStrike's Falcon software uses a special driver that allows it to run at a lower level than most apps so it can detect threats across a Windows system. Microsoft tried to restrict third parties from accessing the kernel in Windows Vista in 2006 but was met with pushback from cybersecurity vendors and EU regulators. However, Apple was able to lock down its macOS operating system in 2020 so that developers could no longer get access to the kernel.

Now, it looks like Microsoft wants to reopen the conversations around restricting kernel-level access inside Windows.
"This incident shows clearly that Windows must prioritize change and innovation in the area of end-to-end resilience," says John Cable, vice president of program management for Windows servicing and delivery, in a blog post titled "Windows resiliency: Best practices and the path forward." Cable calls for closer cooperation between Microsoft and its partners "who also care deeply about the security of the Windows ecosystem" to make security improvements.

Source: https://www.theverge.com/2024/7/26/24206719/microsoft-windows-changes-crowdstrike-kernel-driver

But this does a fair job of refuting why that wouldn't be any better. Basically if Microsoft are the only ones who can access the kernel, when they make a screw up and push the update to everyone (not just those using a piece of software), the impact will be even wider.



There are a number of articles about hospital problems and the global health impacts that alone must have had. There's probably a hundred times more like them that just haven't hit the news yet, or did just in other countries;

Article:

An 83-year-old man has been missing for a week after lingering issues from the CrowdStrike outage canceled his flight home, according to the Orlando Police Department.

Patrick A. Bailey had checked in for his flight out of Orlando International Airport in Florida on July 21. The trip was canceled "due to lingering results from the CrowdStrike outage," police said.

"Bailey spent the night at a local hotel and checked out the following morning," the police department said on X. "However, he did not take a flight home."


No one has heard from Bailey since he checked out of the hotel, police said.

Bailey's son, Patrick Bailey Jr., says his father was meant to fly home to California on July 21 after visiting his sister in Florida, but his flight was canceled, NBC affiliate WESH reported. Patrick said his father left his cellphone at the hotel and he has not been able to reach him since.

Source: https://www.nbcnews.com/news/us-news/83-year-old-man-missing-crowdstrike-outage-canceled-flight-home-rcna163966