Key Takeaways
- Ticket‑based metrics (e.g., closure time, number of tickets) distort SOC behaviour by encouraging analysts to treat alerts as false positives rather than investigating them.
- Measuring the sheer volume of detection rules or log data rewards low‑quality outputs and can erode the value of genuine security coverage.
- The only true indicator of SOC effectiveness is timely detection‑and‑response (TTD/TTR); however, real attacks are rare in a well‑defended environment, making direct measurement difficult.
- Red‑team and purple‑team exercises supplement TTD/TTR by providing realistic, repeatable tests of detection capabilities.
- Treating SOC analysts as subject‑matter experts—granting them deep system knowledge, sufficient time, and clear objectives—maximises the value of hypothesis‑led threat hunting.
- Regularly revisiting and tuning rules that generate excessive false positives prevents analyst fatigue and keeps focus on genuine threats.
- Analyst performance should be assessed on threat knowledge, tool proficiency, and organisational understanding, not merely on ticket throughput.
- Sustained analyst morale is a cultural health indicator; low morale signals management or process issues that need remediation.
Misguided Ticket‑Based Metrics
Many organisations initially try to gauge the success of their Security Operations Centre (SOC) using the same performance indicators applied to IT service desks—such as the average time to close a ticket or the total number of tickets resolved per shift. Dave Chismon, CTO for Architecture at the UK’s National Cyber Security Centre (NCSC), warns that this approach is fundamentally flawed. Ticket‑centric metrics were designed for routine request fulfilment, not for the unpredictable, adversarial nature of cyber‑security work. When applied to a SOC, they create a false sense of productivity while actually diverting effort away from the core mission: detecting and responding to genuine attacks.
Perverse Incentives of Ticket Closure
If SOC analysts are evaluated primarily on how quickly they close tickets, a powerful perverse incentive emerges. Analysts learn that the fastest way to meet their targets is to label an alert as a false positive and close the ticket without thorough investigation. This behaviour reduces the workload in the short term but leaves real threats undetected, increasing the organisation’s risk exposure. Chismon stresses that such a metric system actively works against the SOC’s purpose by rewarding superficial handling of alerts rather than diligent analysis.
Detection Rule Proliferation
A similar problem arises when organisations measure SOC performance by the number of detection rules written. The incentive to increase this count encourages analysts to create many low‑specificity rules that generate high volumes of alerts, most of which are noise. Rather than improving detection quality, this practice inflates the alert queue, overwhelms analysts, and dilutes the signal‑to‑noise ratio. Consequently, the SOC becomes less effective at spotting the subtle indicators of compromise that truly matter.
Log Volume versus Log Quality
Relying on the sheer volume of logs collected as a proxy for coverage suffers from the same short‑sightedness. More logs do not automatically translate into better security; they may simply represent redundant or irrelevant data. When SOCs are judged on log volume, there is temptation to retain logs for shorter periods to keep storage costs low, inadvertently discarding valuable historic data that could be crucial for forensic analysis or trend identification. Chismon argues that quality—relevance, timeliness, and contextual enrichment—must trump quantity when assessing log‑based capabilities.
True Effectiveness: TTD/TTR
The only metric that genuinely reflects whether a SOC is fulfilling its mission is the speed with which it detects and responds to real attacks, commonly expressed as “time to detect” (TTD) and “time to respond” (TTR). A low TTD/TTR indicates that the SOC’s monitoring, alerting, and incident‑response processes are working in concert to limit an adversary’s dwell time. However, measuring TTD/TTR directly presents a challenge: in organisations with strong defence‑in‑depth postures, successful breaches are intentionally rare events, so opportunities to observe these metrics in production are scarce.
Challenges Measuring Rare Attacks
Because genuine attacks may be infrequent, relying solely on observed TTD/TTR can lead to misleading conclusions—a SOC might appear effective simply because it has not been tested, not because its capabilities are sound. Chismon notes that this measurement problem necessitates complementary techniques that can provoke realistic attack scenarios without waiting for an actual intrusion. By deliberately exercising detection and response capabilities, organisations can obtain meaningful data on how quickly their SOC would react under real‑world conditions.
Red Teaming and Purple Teaming
To bridge the measurement gap, Chismon recommends supplementing TTD/TTR with red‑team and purple‑team exercises. Red‑team operations emulate an adversary’s stealthy, goal‑driven approach, providing a covert test of detection fidelity. Purple‑team activities, where red and blue teams collaborate openly, sacrifice some stealth for transparency, allowing defenders to observe attacker tactics in real time and immediately adjust controls. The time saved by not maintaining secrecy can be reinvested into broadening the scope of attack paths examined, thereby delivering higher overall value to the SOC.
Analyst‑Focused SOC Model
Beyond metrics, the NCSC advocates treating SOC analysts as bona fide experts rather than mere ticket processors. This means granting them deep familiarity with the organisation’s networks, applications, and data flows, ensuring they understand the threat landscape relevant to their environment, and providing access to the high‑quality data needed for proactive hunting. Analysts must also be allotted sufficient uninterrupted time to engage in complex investigative work; constant interruption by low‑value alerts erodes expertise and diminishes effectiveness.
Hypothesis‑Led Threat Hunting
Chismon identifies hypothesis‑led threat hunting as the most valuable activity a SOC can undertake. In this approach, analysts formulate educated guesses about how an adversary might compromise the environment—based on threat intelligence, recent vulnerabilities, or observed anomalies—and then search logs and telemetry for evidence to confirm or refute those hypotheses. Even when a hunt yields no direct findings, the process enriches the analyst’s mental model of attacker techniques, sharpens detection intuition, and often produces actionable hardening recommendations or new detection rules that improve future coverage.
Rule Review and False Positive Management
To sustain the benefits of threat hunting, SOCs must routinely review detection rules that generate excessive false positives. High false‑positive rates consume analyst time, foster alert fatigue, and increase the risk of genuine incidents being overlooked. By periodically tuning or retiring noisy rules—especially those stemming from overly broad detection logic—SOCs preserve analyst bandwidth for meaningful investigations and maintain a higher signal‑to‑noise ratio in their alert queues.
Analyst Competency Metrics
Performance evaluation for SOC analysts should shift from ticket‑centric counts to measures that reflect genuine expertise. Suggested metrics include:
- Threat Knowledge: completeness of internal threat‑intelligence documentation, number of relevant reports read and acted upon.
- Tool Proficiency: specific training modules completed, certifications earned, and demonstrated proficiency in SIEM, EDR, network‑traffic analysis, and forensic tools.
- Organisational Understanding: quality of documentation covering critical assets, data flows, and privileged access; strength of relationships with IT administrators and system owners, often measured through joint‑exercise feedback or peer reviews.
These indicators collectively capture an analyst’s ability to anticipate, detect, and mitigate threats in a context‑aware manner.
Culture and Morale
Finally, the NCSC highlights analyst satisfaction as a vital barometer of SOC health. Persistently low morale or high turnover frequently signals deeper cultural or managerial problems—such as unrealistic workloads, insufficient recognition, or inadequate professional development opportunities. Addressing these root causes not only improves retention but also sustains the expertise and motivation required for high‑quality threat hunting, effective incident response, and continuous improvement of detection capabilities.
Conclusion
Re‑evaluating how a SOC is measured is essential to prevent well‑intentioned metrics from undermining its core security mission. By moving away from ticket‑based, volume‑driven indicators and focusing on timely detection‑and‑response, realistic adversarial testing, and the cultivation of expert analysts, organisations can build SOCs that truly protect against evolving cyber threats. The combination of rigorous performance metrics, hypothesis‑led hunting, disciplined rule management, and a supportive analyst‑centric culture forms a resilient foundation for effective security operations in today’s threat‑rich landscape.