CrowdStrike is "deeply sorry" for the "perfect storm of issues" that saw its faulty software update crash millions of Windows machines, leading to the grounding of thousands of planes, passengers stranded at airports, the cancellation of surgeries, and disruption to emergency services hotlines among many more inconveniences.
That apology came on Tuesday when CrowdStrike's senior VP for counter adversary operations, Adam Meyers, appeared before a US House of Representatives cyber security subcommittee hearing about the global IT mess CrowdStrike made.
CEO George Kurtz had earlier declined the invitation to testify. This meant Meyers had the unenviable task of trying to explain what went wrong, and what the security vendor is doing to ensure it never happens again.
Meyers recounted already-known facts about the July 19 incident – namely its origins in the publication of a fresh threat detection configuration content update to CrowdStrike's Falcon endpoint security sensors for Microsoft Windows devices.
"We release 10 to 12 of these content updates every single day," he told lawmakers.
The "perfect storm" Meyers described in his written testimony [PDF] came about due to the update having a "mismatch between input parameters and predefined rules."
The senior veep tried to offer a non-technical explanation of what went wrong, as follows:
Meyers promised that CrowdStrike now pays more attention to the quality of content updates, and uses a phased approach to rollouts of threat-detection updates – which means customers don't have to implement them ASAP.
- House to grill CrowdStrike exec on epic IT meltdown... no, not the CEO
- 1 in 10 orgs dumping their security vendors after CrowdStrike outage
- Post-CrowdStrike catastrophe, Microsoft figures moving antivirus out of Windows kernel mode is a good idea
- CrowdStrike's meltdown didn't dent its market dominance … yet
Kernel access or user mode?
Lawmakers probed the issue of whether it is appropriate for products like CrowdStrike's to enjoy kernel-level access to Windows – as it was that access that meant the bad update was able to crash Windows.
Meyers responded by warning its wares may become less effective without kernel access. Today, he argued, security products like Falcon "have visibility into everything happening on that operating system."
"You can provide enforcement, in other words, threat prevention, and ensure anti-tampering."
This level of tampering, Meyers noted later during the hearing, is a favorite pastime of Scattered Spider – the notorious gang that was behind the Las Vegas casino network intrusions last summer.
Scattered Spider, he warned, has been "using new techniques to elevate their privilege in order to disable security tools on a regular basis," adding that "In order to stop that from happening, we will continue to leverage the architecture of the operating system."
But as Tom Gann, chief public policy officer at threat detection software vendor Trellix, told The Register after the House subcommittee hearing: "Doing these kinds of updates 10 times a day into the kernel, by definition, is just more risky."
Trellix does some kernel updates – but once a quarter, according to Gann.
"Certain types of technical updates and configurations really do need to be done in the kernel," Gann explained. "It's just when we do it, we do it in a very careful, phased approach with a lot of customer oversight. The other work we do is done in user mode."
Microsoft is increasingly fond of user mode. The software giant's response to the CrowdStrike incident has seen it ponder moving antivirus and other threat-detection updates into user mode to reduce the likelihood of major incidents. ®