Fugu-Cyber: Sakana AI’s Orchestration Model Achieves 86.9% on CyberGym and 72.1% on CTI-REALM

Key Takeaways

• Sakana AI introduced Fugu‑Cyber (fugu‑cyber-v1.0), a cybersecurity‑specialized model that serves as the third endpoint in its Fugu orchestration family.
• The model achieved 86.9% success on CyberGym and 72.1% on CTI‑REALM, results Sakana claims are comparable to leading cyber‑focused frontier models such as GPT‑5.5‑Cyber and Claude Mythos Preview.
• CyberGym evaluates an agent’s ability to generate a proof‑of‑concept exploit from a vulnerability description and unpatched code, requiring a validation step that resists gaming.
• CTI‑REALM measures detection‑engineering skill: mapping MITRE ATT&CK techniques, exploring telemetry, refining KQL queries, and producing validated Sigma rules across Linux, AKS, and Azure cloud environments.
• Compared to prior published results, Fugu‑Cyber’s CyberGPM score is a modest improvement over the current frontier (≈85–86%), while its CTI‑REALM score substantially exceeds the top Claude‑based configurations reported by Microsoft.
• The CTI‑REALM metric is a trajectory reward (0‑1 scale); Sakana’s reference to it as a “success rate” is a simplification that warrants caution when interpreting the figure.
• Together, the two benchmarks cover the full security workflow: finding and proving a bug (CyberGym) and turning threat intelligence into actionable detections (CTI‑REALM).

Introduction to Fugu‑Cyber
Sakana AI has expanded its Fugu orchestration suite with a new model explicitly tuned for security reasoning: Fugu‑Cyber (model ID fugu‑cyber‑v1.0). Released just a month after the debut of the broader Fugu orchestrator, this third endpoint is designed to ingest security‑related prompts and produce outputs that assist in vulnerability analysis, exploit development, and threat‑detection engineering. By positioning Fugu‑Cyber as a specialized node within an orchestration framework, Sakana aims to let users combine its cyber‑reasoning strengths with other models for tasks such as code generation, planning, or natural‑language interaction, thereby creating a more versatile AI‑assisted security workflow.

Benchmark Performance Overview
Sakana reports that Fugu‑Cyber attains an 86.9% success rate on the CyberGym benchmark and a 72.1% score on CTI‑REALM. The company frames these numbers as being on par with other cyber‑focused frontier models, notably GPT‑5.5‑Cyber and Claude Mythos Preview. While the raw percentages look impressive, understanding what each benchmark actually measures is essential for interpreting the results correctly.

What CyberGym Measures
CyberGym, developed by researchers at UC Berkeley, consists of 1,507 real‑world vulnerabilities drawn from 188 OSS‑Fuzz projects. In its primary task, an agent receives a textual description of a vulnerability and the corresponding unpatched source code. The goal is to write a proof‑of‑concept (PoC) exploit that crashes the pre‑patch build but leaves the post‑patch build unaffected. A crucial validation step—running the PoC against both builds—prevents agents from simply memorizing known exploits or using trivial crash‑inducing inputs, making the benchmark resistant to gaming. Success is therefore a genuine test of an model’s ability to reason about code, locate the flaw, and synthesize a working exploit.

What CTI‑REALM Measures
CTI‑REALM is an open‑source detection‑engineering benchmark released by Microsoft. It curates 37 public threat reports from sources such as Datadog Security Labs, Palo Alto Networks, and Splunk. An agent must:

1. Map the described adversary behavior to MITRE ATT&CK techniques;
2. Explore relevant telemetry data (logs, network traces, etc.);
3. Iteratively refine Kusto Query Language (KQL) statements to detect the behavior; and
4. Emit validated Sigma rules that can be deployed across Linux endpoints, Azure Kubernetes Service (AKS), and Azure cloud workloads.
   Scoring aggregates performance across these three environments, producing a trajectory reward that ranges from 0 to 1, reflecting the quality and completeness of the detection pipeline constructed by the agent.

Contextualizing the CyberGym Score
When the CyberGym leaders first published their results, the best agent‑model pairing achieved roughly 20%. Subsequent releases showed rapid progress: Anthropic’s Claude Mythos Preview (under Project Glasswing in April 2026) reported 83.1%, while OpenAI’s updated GPT‑5.5‑Cyber reached 85.6% (versus 81.8% for the base GPT‑5.5). Sakana’s 86.9% therefore represents a small incremental step beyond the current frontier rather than a revolutionary leap. It confirms that Fugu‑Cyber is competitive with the leading models but does not dramatically outperform them on this particular exploit‑generation task.

Contextualizing the CTI‑REALM Score
CTI‑REALM’s scoring scheme differs from a simple pass/fail metric; it is a trajectory reward between 0 and 1 that captures the quality of the detection‑engineering process. Microsoft’s own evaluation placed the top three Claude‑based configurations in a narrow band from 0.624 to 0.685. Sakana’s reported 72.1% (0.721) sits above that band, suggesting a stronger performance on this benchmark. However, because Sakana labels the figure a “success rate,” readers should note the nuance: the value reflects a trajectory reward, not a binary success/failure count. Direct comparison with pass/fail rates from other works would require re‑scoring or clarification from Sakana.

Why the Two Benchmarks Together Matter
CyberGym and CTI‑REALM occupy opposite ends of a typical security workflow. CyberGym tests the offensive side—identifying a vulnerability and proving it with an exploit. CTI‑REALM tests the defensive side—transforming threat intelligence into concrete detection rules. By presenting results on both, Sakana highlights that Fugu‑Cyber can contribute to both finding and proving bugs and turning intel into actionable detections. This dual capability aligns with the orchestrator’s vision of chaining models: a security analyst could first use Fugu‑Cyber to generate a PoC, then hand the vulnerability details to another model (or the same model in a different mode) to craft detection rules, all within a single orchestrated pipeline.

Implications for the Security AI Landscape
The release of Fugu‑Cyber signals a growing trend toward domain‑specialized large language models that are embedded in larger AI systems rather than offered as standalone chatbots. By anchoring its performance to well‑known, rigorous benchmarks, Sakana provides the security community with a transparent basis for evaluating whether a model truly adds value over existing options. The modest but meaningful gains on CyberGym, combined with a notable lead on CTI‑REALM, suggest that Fugu‑Cyber may be particularly strong at the reasoning and translation steps required for detection engineering, while remaining competitive on exploit synthesis.

Considerations and Caveats
Readers should keep several caveats in mind when interpreting the reported numbers:

1. Benchmark evolution – Both CyberGym and CTI‑REALM are living benchmarks; future updates could shift baselines.
2. Scoring interpretation – CTI‑REALM’s trajectory reward is not directly comparable to a simple success percentage; Sakana’s labeling may oversimplify.
3. Environmental dependencies – Performance can vary with the specific tooling, prompting strategies, and compute resources used during evaluation.
4. Generalisability – High scores on curated benchmarks do not guarantee real‑world effectiveness across all software ecosystems or threat landscapes.

Conclusion
Fugu‑Cyber represents Sakana AI’s latest effort to push the frontier of AI‑assisted cybersecurity. Its placement as a third endpoint in the Fugu orchestrator underscores a modular approach where specialized models can be summoned for specific subtasks of a security operation. While its CyberGym score marks a modest improvement over the current state‑of‑the‑art, its CTI‑REALM result indicates a stronger capability in turning threat intelligence into detection artifacts. Together, these outcomes suggest that Fugu‑Cyber can be a valuable component in an AI‑driven security pipeline, provided users remain aware of the nuances behind the benchmark metrics and continue to validate model outputs in their own environments.