Not long after Windows PCs and servers at the Australian limb of audit and tax advisory Grant Thornton started BSODing last Friday, senior systems engineer Rob Woltz remembered a small but important fact: When PCs boot, they consider barcode scanners no differently to keyboards.
That knowledge nugget became important as the firm tried to figure out how to respond to the mess CrowdStrike created, which at Grant Thornton Australia threw hundreds of PCs and no fewer than 100 servers into the doomloop that CrowdStrike's shoddy testing software made possible.
All of Grant Thornton's machines were encrypted with Microsoft's BitLocker tool, which meant that recovery upon restart required CrowdStrike's multi-step fix and entry of a 48-character BitLocker key.
The firm prioritized recovery for its servers, and tackled that task manually. But infrastructure manager Ben Watson and Woltz felt the sheer number of PCs at the firm meant an automated response would be required.
That response could not, however, involve distributing BitLocker keys – doing so was just too risky to contemplate.
So was reading keys to workers over the phone or in person. "It felt like a bad idea to read a 48-character key to people who were already stressed out," Woltz told The Register.
Which was when his memory about barcode scanners came into play. The firm had the BitLocker keys for all its PCs, so Woltz and colleagues wrote a script that turned them into barcodes that were displayed on a locked-down management server's desktop. The script would be given a hostname and generate the necessary barcode and LAPS password to restore the machine.
Woltz went to an office supplies store and acquired an off-the-shelf barcode scanner for AU$55 ($36).
At the point when rebooting PCs asked for a BitLocker key, pointing the scanner at the barcode on the server's screen made the machines treat the input exactly as if the key was being typed. That's a lot easier than typing it out every time, and the server's desktop could be accessed via a laptop for convenience.
Woltz, Watson, and the team scaled the solution – which meant buying more scanners at more office supplies stores around Australia.
- Uncle Sam opens probe into CrowdStrike turbulence at Delta Air Lines
- How did a CrowdStrike file crash millions of Windows computers? We take a closer look at the code
- Windows Patch Tuesday update might send a user to the BitLocker recovery screen
- EU gave CrowdStrike the keys to the Windows kernel, claims Microsoft
On Monday, remote staff were told to come to the office with their PCs and visit IT to connect to a barcode scanner. All PCs in the firm's Australian fleet were fixed by lunchtime – taking only three to five minutes for each machine.
Watson told us manually fixing servers needed about 20 minutes per machine.
A Grant Thornton Australia IT worker scanning a laptop displaying the necessary BitLocker barcodes ... Click to enlarge
Woltz is pleased that his idea translated into a swift recovery, but also a little regretful he didn't think of using QR codes – they could have encoded sufficient data to automate the entire remediation process.
Watson thinks Woltz did more than enough. On LinkedIn he hailed the effort of Woltz and other team members as "remarkable innovation in streamlining workstation recovery."
Woltz told The Register he and the team are chuffed that they were able to help, and also that some of them feature as hand models wielding barcode scanners in Watson's LinkedIn post... ®