Analysis Last week, at 0409 UTC on July 19, 2024, antivirus maker CrowdStrike released an update to its widely used Falcon platform that caused Microsoft Windows machines around the world to crash.
The impact was extensive. Supply chain firm Interos estimates 674,620 direct enterprise customer relationships of CrowdStrike and Microsoft were affected. Microsoft said 8.5 million Windows machines failed. The results beyond a massive amount of IT remediation time included global flight and shipping delays due to widespread Windows system failures.
The cause, to the extent so far revealed by CrowdStrike, was "a logic error resulting in a system crash and blue screen (BSOD) on impacted systems."
That crash stemmed from quite possibly mangled data that somehow found its way into a Falcon configuration file called a Channel File, which controls the way CrowdStrike's security software works.
Channel Files are updated over time by CrowdStrike and pushed to systems running its software. In turn, Falcon on those machines uses information in the files to detect and respond to threats. This is part of Falcon's behavioral-based mechanisms that identify, highlight, and thwart malware and other unwanted activities on computers.
In this case, a configuration file was pushed to millions of Windows computers running Falcon that confused the security software to the point where it crashed the whole system. On rebooting an affected box, it would almost immediately start up Falcon and crash all over again.
According to CrowdStrike, Channel Files on Windows machines are stored in the following directory:
C:\Windows\System32\drivers\CrowdStrike\
The files use a naming convention that starts with "C-" followed by a unique identifying number. The errant file's name in this case started with "C-00000291-", followed by various other numbers, and ended with the .sys extension. But these are not kernel drivers, according to CrowdStrike; indeed, they are data files used by Falcon, which does run at the driver level. That is to say, the broken configuration file was not a driver executable but it was processed by CrowdStrike's highly trusted operating-system-level code, and when the bad file caused that code to go off the rails, it brought down the whole operating system – Microsoft Windows in this saga.
"Channel File 291 controls how Falcon evaluates named pipe execution on Windows systems. Named pipes are used for normal, interprocess or intersystem communication in Windows," CrowdStrike explained in a technical summary published over the weekend.
The configuration update triggered a logic error that resulted in an operating system crash
"The update that occurred at 04:09 UTC was designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks. The configuration update triggered a logic error that resulted in an operating system crash."
Translation: CrowdStrike spotted malware abusing a Windows feature called named pipes to communicate with that malicious software's command-and-control (C2) servers, which typically instruct the malware to perform all sorts of bad things. CrowdStrike pushed out a configuration file update to detect and block that misuse of pipes, but the config file broke Falcon.
While there has been speculation that the error was the result of null bytes in the Channel File, CrowdStrike insists that's not the case.
"This is not related to null bytes contained within Channel File 291 or any other Channel File," the cybersecurity outfit said, promising further root cause analysis to determine how the logic flaw occurred.
Specific details about the root cause of the error have yet to be formally disclosed – CrowdStrike CEO George Kurtz has just been asked to testify before Congress over this matter – though security experts such as Google Project Zero guru Tavis Ormandy and Objective-See founder Patrick Wardle, have argued convincingly that the offending Channel File in some way caused Falcon to access information in memory that simply wasn't present, triggering a crash.
It appears Falcon reads entries from a table in memory in a loop and uses those entries as pointers into memory for further work. When at least one of those entries was not correct or present, as a result of the config file, and instead contained a garbage value, the kernel-level code used that garbage as if it was valid, causing it to access unmapped memory.
That bad access was caught by the processor and operating system, and sparked a BSOD because at that point the OS knows something unexpected has happened at a very low level. It's arguably better to crash in this situation than attempt to continue and scribble over data and cause more damage.
Wardle told The Register the crash dump and disassembly make it clear that the crash arose from trying to use uninitialized data as a pointer – a wild pointer – but further specifics remain unknown.
"We still don’t have the exact reason, though, why the channel file triggered that," he said.
The Register spoke with cybersecurity veteran Omkhar Arasaratnam, general manager of OpenSSF, about how things fell apart.
Arasaratnam said the exact cause remains a matter of speculation because he doesn't have access to the CrowdStrike source code or the Windows kernel.
- CrowdStrike CEO summoned to explain epic fail to US Homeland Security
- CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes
- CrowdStrike Windows patchpocalypse could take weeks to fix, IT admins fear
- Angry admins share the CrowdStrike outage experience
CrowdStrike's Falcon software, he said, has two components: A digitally signed, Microsoft-approved driver called CSAgent.sys and the Channel Files used for updating the software with the latest security information.
"What CrowdStrike did is they essentially have a driver that's signed that then loads a bunch of these channel configurations," said Arasaratnam. "We don't know what the channel configuration file actually entails. It's a combination of what's in the file, as well as how CSAgent.sys interprets that."
Based on one stack trace, he explained, CSAgent.sys gets terminated for performing what's known as a bad pointer dereference. It tried to access memory from the address 0x000000000000009c, which didn't exist.
"It was an area of memory that it shouldn't have had access to," said Arasaratnam.
"Now, the Catch-22 you get into when you have a very low-level program like this, is the kernel overall is supposed to be responsible for the operating system doing many low-level things, including allocating memory," Arasaratnam said.
"So if the kernel is trying, in essence, is trying to access memory that it shouldn't access, the appropriate thing to do, just from an operating system theory perspective, is to assume that none of the memory that's been allocated is safe, because if the kernel doesn't know, who the heck does, and basically halt the system."
The situation was complicated by the way the Windows driver architecture works, Arasaratnam explained.
"The way that it works is that drivers can set a flag called boot-start," he said.
"So normally, if you've got a driver that's acting kind of buggy and causes a failure like this, Windows can auto resume by simply not loading the driver the next time. But if it is set as boot-start, which is supposed to be reserved for critical drivers, like one for your hard drive, Windows will not eliminate that from the startup sequence and will continue to fail over and over and over and over again, which is what we saw with the CrowdStrike failure."
(We believe the reason why Microsoft recommended people reboot affected Windows virtual machines on Azure as many as 15 times to clear the problem is because there was a small chance each time that the errant config file would be automatically updated to a non-broken one before the CSAgent.sys driver started parsing it. After multiple reboots, you would eventually win that race condition.)
Arasaratnam said that beyond that, we won't know how the Channel File update that told CSAgent.sys to do a bad pointer dereference managed to pass quality assurance (QA).
"It seems obvious that something slipped past QA given the frequency with which the crash occurred," he said. "It seems like even a trivial amount of QA would have caught this. This isn't some edge case where it's like one in 1,000 machines, right?"
Arasaratnam said there are several best practices that should have been observed. One is not to run software in kernel mode if you can help it. Second, is to ensure that QA is more thorough. Third, is to do what Google does by deploying incremental Canary releases.
He explained, "One of the techniques employed by Google, which we used when I was there, is to do what's called Canary releases – gradual or slow rollouts – and observe what's occurring rather than crashing what Microsoft estimated were 8.5 million machines." ®