Guidance & Support: Windows Systems Experiencing Boot Loop Due to CrowdStrike Update

Executive Summary

Beazley Security is aware of reports concerning crashes on Windows hosts related to the Falcon Sensor. Beazley Security has written recovery guidance for organizations who are impacted by this issue.

The issue was caused by a single “channel” file update deployed by CrowdStrike on 2024-07-19 04:09 UTC. Any devices running Falcon Sensor that were online at the time of the update crashed and were stuck in a “Blue Screen of Death” (BSOD) boot loop, unable to boot into the Windows Operating System.

CrowdStrike identified the problem and reverted the defective file about ninety minutes later at 05:27 UTC. Any devices that were not online at the time of the update or started after the 05:27 UTC were not impacted.

Update (2024-07-22 15:00UTC) – Microsoft released a recovery tool based on CrowdStrike’s repair scripts. Details can be found here.

Affected Systems / Products

The issue was caused by channel file update C-00000291*.sys which was published on 2024-07-19 04:09 UTC. CrowdStrike reverted this content update at 05:27 UTC.

Any devices operational during the 04:09 UTC update are affected.

Any devices booted after the 05:27 UTC update are not affected.

Additionally, any devices running Windows 7/2008 R2 are not affected.

Mitigations / Workarounds

Update (2024-07-22 15:00 UTC) Microsoft Recovery Tool

Microsoft has released a recovery tool to assist the repair process. The tool can be downloaded here, with a detailed writeup found here. The tool will create boot media that then provides two options:

Recover from WinPE – Directly recover a system without local admin privileges. If BitLocker is present the BitLocker recovery key must be manually entered. Systems using third-party disk encryption will need to recover the drive before using this option.

Recover from safe mode – Boot into safe mode, then run remediation steps. This requires local administrator privileges. Use this approach for devices using TPM-only protectors, devices that are not encrypted, or situations where the BitLocker recovery key is unknown. However, if utilizing TPM+PIN BitLocker protectors, the user will either need to enter the PIN if known, or the BitLocker recovery key must be used.

To create the boot media, users will need:

A Windows 64-bit client with at least 8GB of free space.
Administrative privileges on the Windows client from prerequisite #1.
A USB drive with min 1GB and max of 32GB.

On this machine, users will perform the following steps:

Download the signed Microsoft Recovery Tool from the Microsoft Download Center.
Extract the PowerShell script from the downloaded solution.
Run MsftRecoveryToolForCSv2.ps1 from an elevated PowerShell prompt.
The ADK will download and media creation will start. It may take several minutes to complete.
Choose one of the two options mentioned above for recovering affected devices (see additional details below).
Optionally select a directory that contains driver files to import into the recovery image. Keyboard and mass storage drivers may be needed. Network or other drivers are not required. Microsoft recommends you select “N” to skip this step. The tool will import any SYS and INI recursively under the specified directory.
Select the option to either generate an ISO or USB drive and specify drive letter.

Once the boot media is created here are the recovery steps for each chosen option.

For recovery from WinPE media:

Insert the USB key into an impacted device.
Reboot the device.
During restart, press F12 (or follow manufacturer-specific instructions for booting to BIOS).
From the BIOS boot menu, choose Boot from USB and continue.
The tool will run.
If BitLocker is enabled, the user will be prompted for the BitLocker recovery key including the dashes. The recovery key options are provided here. For third-party device encryption solutions, follow any steps provided by the vendor to gain access to the drive.
The tool will run the issue-remediation scripts as recommended by CrowdStrike.
Once complete, remove the USB drive and reboot the device normally.

For recovery from Safe Boot:

Insert the USB key into an impacted device.
Reboot the device.
During restart, press F12 (or follow manufacturer-specific instructions for booting to BIOS).
From the BIOS boot menu, choose Boot from USB and continue.
The tool runs.
The following message appears: "This tool will configure this machine to boot in safe mode. WARNING: In some cases you may need to enter a BitLocker recovery key after running."
Press any key to continue.
The following message appears: "Your PC is configured to boot to Safe Mode now."
Press any key to continue.
The machine reboots into safe mode.
The user runs repair.cmd from the root of the media/USB drive. The script will run the remediation steps as recommended by CrowdStrike.
The following message appears: "This tool will remove impacted files and restore normal boot configuration. WARNING: You may need BitLocker recovery key in some cases. WARNING: This script must be run in an elevated command prompt."
Press any key to continue.
The user repair will run and the normal boot flow will be restored.
Once successful, the user will see the following message: “Success. System will now reboot.”
Press any key to continue. The device will reboot normally.

There are further steps and details for situations such as systems on Hyper-V, or using PXE recovery. Those details can be found on their instructions here.

Organizations impacted by this issue should attempt to reboot the affected systems several times to see if they’re able to re-connect to CrowdStrike and download a reverted channel file. If the affected system continues to see issues, organizations will need to leverage Windows Safe Mode for manual recovery. Please review the recovery steps below for more details.

Manual Recovery of a system by deleting the CrowdStrike channel file

Cycle through several “Blue Screen of Death” (BSOD)s until you get to a recovery screen

At the recovery screen navigate to Troubleshoot -> Advanced Options -> Command Prompt. If you are prompted for a BitLocker recovery key, select the option to “Skip this drive”

Once you are in the “Command Prompt”, we will enter the following command to force Windows to reboot into Safe Mode in order to manually remove the affected channel update file:

bcdedit /set {default} safeboot minimal

Once the command above has been successfully entered into the Command prompt, reboot the device to have it automatically boot into Safe Mode

Once the device reboots into Safe Mode, you should be able to log in normally the machine. Log into the machine with a Local Administrator account.

Open another Command Prompt and enter the following commands to delete the channel file:

cd “C:\Windows\System32\drivers\CrowdStrike”

del C-00000291*

Now enter the following command to ensure the computer boots normally in the future (i.e not into Safe Mode)

bcdedit /deletevalue {current} safeboot

Once the above commands have been entered, you may reboot the computer. The computer should reboot into Window’s normal operating mode.

Recovery of AWS EC2 Virtual Machines that are impacted

Organizations with AWS EC2 Windows Virtual Machines that are impacted should follow the guidance provided by AWS in order to recover. Organizations with recent backups or snapshots of affected systems may choose to restore a backup of the system from before 2024-07-19 04:09 UTC.

The steps outlined by Amazon are listed below:

Create a snapshot of the EBS root volume of the affected instance

Create a new EBS volume from the snapshot in the same Availability Zone

Launch a new instance in that Availability Zone using a different version of Windows

Attach the EBS volume from step (2) to the new instance as a data volume

Navigate to the \windows\system32\drivers\CrowdStrike\ folder on the attached volume and delete "C-00000291*.sys"

Detach the EBS volume from the new instance

Create a snapshot of the detached EBS volume

Create an AMI from the snapshot by selecting the same volume type as the affected instance

Call replace root volume on the original EC2 Instance specifying the AMI just created

Direct Guidance form AWS is available via their AWS Status Portal here:

https://health.aws.amazon.com/health/status

For guidance on how to attach an Elastic Block Store (EBS) volume to another instance please review the following documentation from Amazon

Recovery of Azure Virtual Machines that are impacted

Organizations with Azure Windows Virtual machines that are impacted should follow the guidance provided by Azure in order to recover. Organizations with recent backups or snapshots of affected systems may choose to restore a backup of the system from before 2024-07-19 04:09 UTC.

Organizations with impacted machines should attempt to recover impacted systems by attaching the OS disk to another non-impacted Windows machine and manually deleting the corrupted channel file. Leverage the following Azure documentation to mount the OS disk of an impacted machine on a working Windows VM:

https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-recovery-disks-portal-windows

Once the disk has been mounted on a working system, navigate to the following location on the mounted drive: ‍

\windows\system32\drivers\CrowdStrike\

From the location above, delete the file matching the following pattern:

C-00000291*.sys

Once the file above has been deleted, detach the disk from the working Windows system and attempt to boot the impacted Virtual Machine normally. For more guidance from Azure please review their status page.

Patches

A fix for the corrupted “channel” update file was released 2024-07-19 05:27 UTC and confirmed by the CrowdStrike CEO at 09:45 UTC. Users with affected systems will need to follow the manual recovery steps detailed above to restore their machines to a state where the updated (non-corrupted) channel file can be automatically downloaded and applied.

Attempts to capitalize on the outage

Several Threat Actors have already begun registering domains with the CrowdStrike brand in an attempt to mislead users or in phishing attempts. Beazley Security strongly recommends that organizations only connect to their CrowdStrike Falcon Consoles via a bookmarked URL or to navigate to CrowdStrike’s support portal at the following URL: https://supportportal.crowdstrike.com

Beazley Security has also seen reports that opportunistic Threat Actors are calling organizations known to be CrowdStrike clients and pretending to be CrowdStrike Support attempting to aid impacted end users.

Technical Details

The outage was due to a defective content update for Windows hosts. Mac and Linux hosts were not impacted. The specific “channel” file that caused the issue was C-00000291*.sys, sent on 2024-07-19 04:09 UTC. These channel update files are retrieved and applied automatically and are separate from Falcon sensor updates.

Impacted users reported machines experiencing a system crash with the error message:

csagent.sys (PAGE_FAULT_IN_NONPAGED_AREA)

These machines were then stuck in recovery mode. Restoring machines in this state requires the manual steps detailed above.

How Beazley Security is Responding

Beazley Security is actively monitoring client systems and providing support to implement necessary workarounds and patches. For further assistance, contact Beazley Security.