MSP Guidance on How to Address the Global Tech Outage

MSPs globally woke up to a nightmare on Friday. Late Thursday evening Microsoft users across the globe started experiencing Windows Blue Screen of Death (BSOD) errors and/or reboot loops due to a third-party update from cybersecurity vendor Crowdstrike. This outage is impacting systems across numerous industries including banking, airlines, medical, government, manufacturing, and more. Crowdstrike has issued a statement that they have resolved the cause of the issue. Although this limits or mitigates the spread of the problem it still leaves many managed systems in a down state.

To complicate matters, in addition to the boot issues impacting many users, Microsoft is experiencing outages in its Azure and Office 365 services that are related to this same Crowdstrike update as well. You can track current updates from Microsoft on X (formerly Twitter) by following @MSFT365Status or navigating to Microsoft 365 Service health status.

Unfortunately, you can’t use any RMM tool or any remote access tool to mitigate this issue. The team at Syncro is available to help our partners in any way we can, and we have researched and summarized the steps to remediate to help save you some time.

Mitigation

What you can do to mitigate if rebooting the system does not resolve the issue:

• Boot Windows into Safe Mode or the Windows Recovery Environment
• Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
• Locate the file matching “C-00000291*.sys”, and delete it
• Boot the host normally

If you are able to boot an impacted computer into safe mode you can use Screen Connect or Splashtop to connect remotely to the asset. If you have configured your RMM configured to run in safe mode you can script the removal of the C-00000291*.sys file.

We are also seeing emerging threats of people impersonating Crowdstrike or ways to mitigate. Please be conscious of using third parties that you have never used previously.

If you are running a Virtual Machine in Azure, Microsoft has released steps for how to repair your OS disk offline by following the following instructions

• Attach an unmanaged disk to a VM for offline repair

If your disk is encrypted you will need to follow these additional steps:

• Unlocking an encrypted disk for offline repair

Once you have accessed the disk you will need to follow the original steps for deleting the “C-00000291*.sys” file.

If you are unable to get the machine into Safe Mode we recommend using your BCDR to virtualize in the cloud or on a local appliance or complete a full BMR.  You will need to choose a recovery point from before 19:00 UTC on the 18th of July.

Acronis:

• The Ultimate Recovery Drive: The Acronis Survival Kit
• 30047:Acronis Backup for VMware: Bare Metal Recovery of ESXi Hosts

Datto BCDR:

• Starting a local virtualization on a Datto appliance
• USB Bare Metal Restore (BMR): Getting Started
• Datto Endpoint Backup for PCs Bare Metal Restore: Getting started
• Datto Endpoint Backup for PCs: Virtualization

Veeam

• Performing Restore – Veeam Agent for Microsoft Windows Guide

Axcient

• BMR (Bare Metal Restore) Guide – Axcient
• Virtual Office – Getting started – x360Recover (VO)
• 4. Recover with D2C – Axcient

Unitrends

• Recovery Overview
• Windows Bare Metal Protection and Recovery

Cove

• Recovery
• Restore
• Cove: How do I perform a Bare Metal Restore?

Barracuda Intronis

• Microsoft Windows System Recovery | Barracuda Campus

 

If you are a Syncro customer and need any assistance – please reach out to our support team or reach us at 1-415-523-6363 and we are happy to help in any way we can.

Enjoy this article? Share it!