On July 19, 2024, a faulty update to CrowdStrike’s Falcon Sensor software inadvertently led to the infamous Blue Screen of Death (BSOD) for approximately 8.5 million Windows workstations globally, rendering them unable to reboot.  The disruption was caused by a faulty csagent.sys file. It affected critical services worldwide, including healthcare, finance, and emergency services.  This incident highlighted the vulnerabilities in our interconnected digital infrastructure and emphasized the need for robust disaster recovery plans.

Challenges Faced by Companies

Despite the release of a patch, companies faced significant challenges in accessing their workstations to apply it:

1. Inaccessible Workstations: Workstations caught in a BSOD loop could not be accessed commonly.
2. Manual Intervention: The fix required manual steps, complicating the recovery process for large numbers of workstations.
3. Security Concerns: Bypassing security features to apply the fix raised additional concerns.

Response and Mitigation Strategies

In response to the disruption, experts provided several manual workarounds and strategies to help users stabilize their workstations and restore functionality, particularly when companies couldn’t access their workstations to implement the patch. These methods addressed the issues caused by the faulty csagent.sys file (C-00000291.sys), allowing organizations to repair their workstations efficiently by deleting the faulty C-00000291.sys file.  This approach ensured that affected companies could quickly recover and resume normal operations.

On July 22, 2024, Microsoft released an updated recovery tool featuring two repair options to help IT administrators streamline the repair process.  The tool automates the process of deleting the faulty C-00000291.sys file from Windows workstations, servers, and Hyper-V virtual machines (VM)

Note: For additional recovery strategies, refer to Remediation and Guidance Hub: Falcon Content Update for Windows Hosts

Using the Microsoft Recovery Tool

The Microsoft Recovery Tool includes two repair options:

• Recover from Windows Preinstallation Environment (PE): This option uses boot media to automate the workstation repair process.
• Recover from Safe Mode: This option uses boot media to boot affected workstations into Safe Mode.  An administrator can then sign in with an account with local administrative privileges and run the necessary remediation steps.

Recovering from Windows PE

This option recovers workstations without requiring local administrative privileges.  If the workstation uses BitLocker, you may need to manually enter the BitLocker recovery key before repairing the affected workstation.

Refer to the vendor’s guidance if you use a non-Microsoft disk encryption solution.

Creating boot media

You need the following prerequisites to create the boot media:

• Windows 64-bit Workstation: Ensure you have a Windows 64-bit workstation with at least 8 GB of free space to run the tool and create the bootable USB drive.
• Administrative Privileges: You need administrative privileges on the Windows Workstation mentioned in prerequisite #1.
• USB Drive: Use a USB drive with a minimum size of 1 GB and a maximum size of 32 GB.  The tool will delete all existing data on this drive and automatically format it to FAT32.

To create recovery media:

1. Download the signed Microsoft Recovery Tool from the Microsoft Download Center.
2. Extract the PowerShell script from the downloaded file.
3. Run Windows PowerShell as an administrator and execute the following script: MsftRecoveryToolForCS.ps1.
   
   The Windows Assessment and Deployment Kit (Windows ADK) is downloaded and installed.
   Note: This process might take several minutes to complete.
   
4. Select the Windows PE option.
5. (Optional) Select a directory containing driver files to import into the recovery image, or select N to proceed (recommended.)
6. Select ISO file or USB drive.
   If USB drive is selected:
   1. Insert the USB drive when prompted and provide the drive letter.
   2. After the tool completes creating the USB drive, safely remove it from the Windows workstation.

Using the Boot Media for Windows PE recovery option

You need the following prerequisites to use the boot media:

• You may need the BitLocker recovery key for each BitLocker-enabled and affected workstation.
• If the affected workstation uses Trusted Platform Module (TPM) and PIN protectors and you don’t know the PIN for the workstation, you require the recovery key.
  

To use the boot media for Windows PE recovery:

1. Insert the USB key into an affected workstation.
2. Restart the workstation.
3. During restart, press F12 to access the BIOS boot menu.
   
   Note: Some workstations may use a different key combination to access the BIOS boot menu.  Follow manufacturer-specific instructions for the workstation.
   
4. From the BIOS boot menu, select Boot from USB and continue.
   The Microsoft Recovery Tool runs.
   If BitLocker is enabled, the Microsoft Recovery Tool prompts the user for the BitLocker recovery key.
5. Enter the BitLocker recovery key, including the dashes (-).
   1. If BitLocker isn’t enabled on the workstation, you may still be prompted for the BitLocker recovery key.  Press Enter to skip and continue.

The Microsoft Recovery Tool runs the remediation steps as recommended by CrowdStrike.

Note: For non-Microsoft workstation encryption solutions, follow the steps provided by the vendor.

For more information on recovery key options, see Where to look for your BitLocker recovery key.

6. After completion, remove the USB drive and restart the workstation normally.

Recovering from Safe Mode

This option to recover from Safe Mode may enable recovery on BitLocker-enabled workstations without requiring the entry of BitLocker recovery keys.  You will need access to an account with local administrator rights on the workstation.

Use this option for workstation in the following situations:

• The workstation uses TPM-only protectors.
• The disk isn’t encrypted.
• The BitLocker recovery key is unknown.

Important:

If the workstation uses TPM+PIN BitLocker protectors, the user will either need to enter the PIN, or you will need to use the BitLocker recovery key.

If BitLocker isn’t enabled, the user only needs to sign in with an account with local administrator rights.

Note: For non-Microsoft workstation encryption solutions, follow the steps provided by the vendor.

See Creating boot media for instructions on creating boot media.

Using the Boot Media for Safe Mode recovery option

You need the following prerequisites to use the boot media:

• Access to the local Administrator account.
• If the affected workstation uses Trusted Platform Module (TPM) and PIN protectors and you don’t know the PIN for the workstation, you require the recovery key.

To use the boot media for Safe Mode recovery:

1. Insert the USB key into an affected workstation.
2. Restart the workstation.
3. During restart, press F12 to access the BIOS boot menu.
   
   Note: Some workstations may use a different key combination to access the BIOS boot menu.  Follow manufacturer-specific instructions for the workstation.
4. From the BIOS boot menu, select Boot from USB and continue.
   The Microsoft Recovery Tool runs, and the following message appears:
   
   This tool will configure this machine to boot in safe mode.  WARNING: In some cases, you may need to enter a BitLocker recovery key after running.
5. Press any key.
   The following message appears:
   
   Your PC is configured to boot to Safe Mode now.
6. Press any key.
   The workstation restarts into safe mode.
7. Run repair.cmd from the media drive root.
   The script runs the remediation steps as recommended by CrowdStrike.
   The following message appears:
   
   This tool will remove impacted files and restore normal boot configuration.  WARNING: You may need BitLocker recovery key in some cases.  WARNING: This script must be run in an elevated command prompt.
8. Press any key.
   The script runs and restores the normal boot mode.
   After the tool successfully completes, the following message appears:

Success.  System will now reboot.

9. Press any key.
   The workstation restarts normally.
   

Using the recovery media on Hyper-V virtual machines

You can use the recovery media to remediate affected Hyper-V virtual machines (VMs).  When creating boot media, select the option to generate an ISO file.

Note: For non-Hyper-V VMs, follow the instructions provided by your hypervisor vendor for using the recovery media.

To use the recovery media on Hyper-V virtual machines:

1. From Hyper-V settings, select SCSI Controller > DVD Drive
2. Browse to the recovery ISO and add it as an Image file
3. Note the Boot order settings so that you can manually restore them later
4. Change the Boot order and set DVD Drive as the first boot entry.
5. Start the VM and press any key when prompted to boot from the ISO image.
6. Depending on how you created the recovery media, follow the additional steps to use the Windows PE or Safe Mode recovery options.

Conclusion

The CrowdStrike Falcon incident on July 19, 2024, highlighted the critical need for robust recovery strategies.  When a faulty update caused millions of Windows workstations to crash, it exposed vulnerabilities in our digital infrastructure and emphasized the importance of comprehensive disaster recovery plans.

Microsoft’s updated recovery tool, offering both Windows PE and Safe Mode repair options, proved essential in addressing these challenges.  These tools enabled IT administrators to recover workstations and minimize downtime efficiently, even in complex encryption scenarios.

This incident underscores the need for well-prepared recovery plans and access to encryption keys.  By implementing these strategies, organizations can ensure quick operation restoration and maintain continuity in the face of significant cybersecurity threats.