Key Takeaways
- St. Paul’s IT team detected suspicious activity first at the city water utility, which shared a network with municipal systems but maintained its own staff.
- The ransomware group Interlock used a double‑extortion tactic, exfiltrating data before encrypting it; nightly backups allowed the city to refuse paying a ransom.
- Recovery prioritized 911, payroll, and water services, with email, libraries, and payment systems restored by late August and full recovery taking several months.
- Minnesota’s governor activated the National Guard’s cyber unit under an emergency executive order after local capacity was exceeded.
- The Guard’s 177th Cyber Protection Team provided connectivity, manpower, laptop deployment, and enhanced endpoint‑detection tools across city departments.
- Operation Secure St. Paul—a city‑wide, in‑person password reset and device security check at a large arena—re‑authenticated over 3,000 employees in three days.
- Public sharing of the after‑action story, collaboration with peer cities, and participation in conferences aim to strengthen statewide cyber preparedness.
- Ongoing efforts include maintaining a Digital Security Incident Info Hub, refining incident‑response plans, and leveraging pre‑existing relationships for faster future coordination.
Detection and Initial Response
The ransomware incident was first spotted by the IT team operating St. Paul’s water utility, which, although part of the city, maintains its own technology staff and systems while sharing a common network with municipal operations. Chief Information Officer Jaime Wascalus reported that the utility was already using endpoint detection and response (EDR) tools funded through the federal State and Local Cybersecurity Grant Program and administered by Minnesota Information Technology Services (MNIT). When anomalous behavior triggered alerts, the utility’s staff isolated the affected segment and notified the broader city IT division, setting off the city’s incident‑response workflow. Early detection, aided by the EDR solution, allowed officials to begin containment measures before the malware could spread unchecked across all government‑connected devices.
Scope of Impact and Ransomware Details
Once the network was deliberately shut down to halt the malware’s propagation, internal networks, online payment portals, and public Wi‑Fi services were taken offline. The attackers were identified as the ransomware gang Interlock, which employs a double‑extortion model: first exfiltrating sensitive data, then encrypting systems and demanding payment for both decryption and a promise not to leak the stolen information. Despite the pressure, St. Paul leadership opted not to pay the ransom, citing the existence of reliable nightly backups that could restore critical functions without conceding to criminal demands. The decision underscored the value of robust backup strategies as a deterrent to extortion.
City’s Backup Strategy and Decision Not to Pay
St. Paul’s pre‑existing policy of performing nightly backups proved pivotal during the crisis. When Interlock encrypted files, the city could rely on recent backup copies to rebuild essential services, thereby eliminating the leverage the attackers hoped to gain through data loss. Wascalus emphasized that the backup regimen, combined with a well‑tested restoration process, gave city leaders confidence to refuse the ransom demand. This stance not only avoided financing criminal enterprises but also reinforced a policy stance that paying ransoms often encourages further attacks. The successful reliance on backups has since become a highlighted example in the city’s cybersecurity training materials.
Escalation to State Resources and National Guard Activation
Even with strong internal preparations, the scale of the disruption quickly exceeded the city’s solo response capacity. St. Paul reported the incident through MNIT’s cyber‑incident‑reporting portal and engaged a contracted cybersecurity firm for supplemental expertise. Recognizing that local resources were stretched thin, Governor Tim Walz issued an emergency executive order activating the Minnesota National Guard’s specialized cyber unit. Lt. Col. Brian L. Morgan, director of the Guard’s cyber coordination cell, explained that such state support is reserved for situations where public safety, health, or essential services are imperiled beyond local capability—a threshold met by the St. Paul attack. The Guard’s deployment was thus framed as a measured, last‑resort augmentation rather than a replacement of municipal efforts.
Role of the Minnesota National Guard Cyber Protection Team
The Guard’s 177th Cyber Protection Team, comprising roughly 50 volunteer soldiers and a small cadre of full‑time staff, arrived to bolster St. Paul’s defenses. Their contributions included establishing secure connectivity via FirstNet, deploying additional laptops for response personnel, supplying manpower for monitoring and remediation, and installing enhanced endpoint‑detection solutions across various city departments. Beyond immediate technical aid, the Guard emphasized its ongoing mission to train for ransomware scenarios, conduct threat‑hunting exercises, and protect critical infrastructure. By cultivating relationships with local, state, and federal partners well before incidents occur, the Guard aimed to accelerate coordination and reduce response latency when cyber emergencies arise.
Operation Secure St. Paul: Mass Password Reset and Device Checks
In the weeks following the initial containment, city leaders launched Operation Secure St. Paul—a sweeping, in‑person security overhaul held at a 5,000‑seat arena. The operation required every city employee to appear physically to undergo a global password reset, implement multi‑factor authentication (MFA), and have their devices inspected for approved security software and configurations. Wascalus described the effort as a “huge logistical undertaking” that took about five days to plan, though she noted it ideally should have required months of preparation. Despite the compressed timeline, the initiative successfully re‑authenticated more than 3,000 employees within three days, dramatically reducing the risk of lingering compromised credentials and reinforcing a culture of security hygiene across the workforce.
Collaboration, Information Sharing, and Lessons Learned
St. Paul officials have opted to share their experience publicly, following the precedent set by other jurisdictions such as Dallas and Nevada, which released after‑action reports and presented lessons at conferences. Mayor Melvin Carter disclosed conversations with the mayors of Atlanta and Baltimore—cities that have likewise endured significant cyber attacks—to exchange insights on response and recovery. Wascalus reflected that the incident reshaped her perspective on collaboration, noting that her CIO peer network provided informal support, shared best practices, and even loaned equipment during the crisis. These interactions reinforced the premise that relationships forged before an incident are invaluable assets when a cyber event unfolds.
Ongoing Preparedness and Future Outreach
To institutionalize the lessons gained, St. Paul maintains an online Digital Security Incident Info Hub where documents, timelines, and recommendations are accessible to staff and other municipalities. An after‑action report is currently under review and will soon be finalized for broader distribution. In June, Wascalus and Emergency Management Director Rick Shute are slated to present the city’s story at the League of Minnesota Cities conference, emphasizing a simple message: “This is what we learned. This is how you need to prep. This is what you need to be ready for in the moment.” By continuously refining incident‑response plans, investing in proactive cybersecurity measures, and nurturing intergovernmental partnerships, St. Paul aims to transform a painful episode into a catalyst for stronger, more resilient cyber defenses across Minnesota and beyond.
