Key Takeaways
- A tertiary, off‑site cloud backup (Backblaze) that was completely isolated from the corporate domain saved the organization when primary backups were inaccessible.
- Credentials for the cloud backup were stored in a personal password manager, highlighting the risk‑and‑reward of deviating from policy during emergencies.
- The ransomware group LockBit lowered its demand to ≈ $750 k after realizing the stolen data was only ~2 GB and contained few sensitive records; the organization chose not to pay.
- Recovery costs (fees, deductibles, labor, legal, etc.) were estimated at ≈ $300 k, far below the ransom amount.
- Critical on‑premises services—Active Directory, building automation, security cameras, door‑badge controllers, and lighting—went down, while SaaS‑based student systems remained operational.
- The breach originated from a compromised personal laptop used to VPN into the network; attackers harvested cached credentials, escalated privileges, and created backdoor accounts.
- A 13‑day dwell time (April 1 – mid‑June) preceded data exfiltration; full remediation was achieved by early July.
- Recommendations: maintain a three‑tier backup strategy with at least one tier fully air‑gapped, enable object‑lock immutability, test backups quarterly, use out‑of‑band communication during incidents, involve cyber‑insurance carriers and outside counsel promptly, file an IC3 report, and leverage CISA guidance.
- Healthcare organizations should prioritize resilient backup and identity‑governance practices over paying ransoms to avoid disrupting patient care.
The Role of the Off‑Site Cloud Backup
When the ransomware attack crippled the primary environment, the organization still had a tertiary backup location hosted in the cloud via Backblaze. This backup was deliberately kept off‑site and not tied to the corporate domain or standard user accounts, which prevented the attackers from encrypting or deleting it. Because the credentials to access Backblaze were saved in a personal password manager (a cloud‑based tool), the team was able to log in, pull down the Active Directory database, and begin restoration—even though they had to temporarily load the data onto an eSports gaming PC due to a lack of spare hardware.
Credential Access and Policy Deviations
Reliance on a personal password manager violated internal policy, yet it proved decisive in the crisis. Without those saved Backblaze credentials, the team would have had no way to reach the isolated backup, underscoring the tension between strict security controls and the need for accessible emergency access. The incident prompted a review of how privileged credentials are stored and who may access them under emergency conditions.
Negotiation with LockBit and the Decision Not to Pay
LockBit initially demanded a higher sum but reduced its ransom to roughly $750,000 after realizing the volume of exfiltrated data was modest. The attackers threatened to publish the stolen data on the dark web if payment was not made. After assessing the data—only about 2 GB in size and containing perhaps four Social Security numbers—the organization concluded the leaked information was not highly sensitive and opted not to pay. LockBit followed through on its threat, posting the data, but the limited scope minimized reputational and regulatory fallout.
Financial Impact of the Incident
Although no ransom was paid, the organization incurred substantial recovery expenses. Zach Lewis estimated total out‑of‑pocket costs at ≈ $300,000, covering incident‑response fees, insurance deductibles, overtime for staff diverted from other projects, legal counsel, and related expenses. This figure was well below the ransom amount, reinforcing the financial advantage of investing in robust backups and incident response rather than paying extortionists.
Systems Affected Versus Those That Remained Operational
The ransomware knocked out several critical on‑premises services: the Active Directory domain controller, the internal network, building‑automation systems (heating and cooling), security cameras, door‑badge controllers, and lighting controls. Notably, door schedules remained functional because they were cached locally on each badge reader. In contrast, the institution’s SaaS‑based educational platforms—used by students for classes and homework—were unaffected, allowing learning to continue uninterrupted despite the backend outage.
Active Directory and Password‑Manager Challenges
With Active Directory offline, the team could not create, modify, or delete user accounts, nor reset passwords. The primary backup of the password manager was also unavailable, creating a potential lock‑out scenario. Fortunately, the personal password manager containing the Backblaze login credentials was still accessible, providing a vital lifeline. This highlighted a gap: reliance on a single authentication source for backup access can become a single point of failure during an attack.
Importance of Out‑of‑Band Communication
During the crisis, the organization learned that conventional communication channels (email, internal chat) could be monitored or disrupted by the attackers. Shifting to out‑of‑band (OOB) methods—such as phone calls, SMS, or separate messaging apps—became essential for coordinating response efforts without alerting or aiding the threat actors. The experience cemented OOB communication as a core component of their incident‑response playbook.
Attack Vector and Privilege Escalation
Investigators traced the initial compromise to a personal laptop belonging to an employee that was not managed by the organization’s IT team. The device likely fell victim to a phishing email, harvesting the user’s credentials. Those credentials were then used to connect to the corporate VPN, granting the attackers entry. Once inside, they discovered cached credentials on a server, which they used to escalate privileges, ultimately gaining control of the hypervisor and creating several backdoor accounts. LockBit even supplied a document listing 10 usernames and passwords they had cracked, confirming the depth of their foothold.
Timeline of the Breach and Recovery
The threat actors first entered the network on April 1 and remained undetected for about 13 days. Data exfiltration occurred around June 12, after which the ransom note was delivered. Through diligent restoration efforts—leveraging the isolated Backblaze backup, rebuilding AD, and securing compromised accounts—the organization achieved full operational status by the first week of July. This dwell time emphasized the need for continuous monitoring and rapid detection capabilities.
Why the Cloud Backup Was Decisive
Zach Lewis described the Backblaze backup as the “chef’s kiss” of their recovery strategy. Its physical and logical separation from the production environment ensured that ransomware could not reach it. The episode reinforced the best practice of maintaining at least one backup tier that is completely air‑gapped or otherwise isolated from the primary network and user authentication systems.
Recommendations for a Resilient Backup Strategy
Drawing from the incident, the team advises a three‑tier backup approach:
- Primary on‑site backups for quick restores.
- Secondary off‑site backups (e.g., another cloud region) for geographic diversity.
- Tertiary, isolated backup (such as Backblaze with object‑lock enabled) that cannot be accessed via standard domain credentials.
Regularly testing restores from each tier—quarterly in this case—and documenting the process ensures that any staff member can recover systems even if key personnel are unavailable.
Immutability and Object Lock
Kari Wilson highlighted the value of object lock (immutability) as a simple “toggle” that prevents backed‑up data from being encrypted, altered, or deleted for a defined period. Enabling this feature guarantees a clean, tamper‑proof copy of critical data, turning backups into a true last line of defense. She noted that many organizations either neglect to enable immutability or fail to test those immutable backups, leaving a gap in their resilience posture.
Guidance for Other Victims of Ransomware
For organizations facing a similar attack, Zach Lewis offered the following actionable advice:
- Activate cyber insurance immediately and know the policy’s coverage details; insurers can bring in specialized resources far beyond an internal team’s capacity.
- Engage outside counsel with ransomware experience; general legal teams may lack the necessary expertise for negotiation and regulatory reporting.
- File an IC3 (Internet Crime Complaint Center) report to involve the FBI, which can assist in tracing and potentially recovering ransom payments, especially when cryptocurrency is involved.
- Leverage CISA for free guidance, remediation steps, and best‑practice frameworks.
- Maintain out‑of‑band communication channels to coordinate response without alerting attackers.
- Stay calm and methodical; the situation is stressful, but a disciplined approach improves outcomes.
He also observed that while the industry has not reduced the frequency of attacks, it has improved recovery capabilities, largely due to better backup practices.
Special Considerations for Healthcare
In healthcare settings, the impulse to pay a ransom often stems from a desire to avoid interrupting patient care. Zach Lewis argued that hospitals should instead focus on robust backup strategies, identity governance, and data‑resiliency measures. By ensuring that critical clinical data is stored in immutable, isolated backups and that access controls are rigorously validated, healthcare providers can restore operations quickly without financing criminal enterprises, thereby reducing the pressure on an already strained sector.
This summary captures the essential events, lessons, and recommendations drawn from the organization’s ransomware experience, highlighting how a well‑designed, isolated backup strategy—combined with disciplined incident response—can mitigate both operational and financial damage.