Key Takeaways
- Mythos Preview detects significantly more software vulnerabilities than prior AI models, confirming Anthropic’s primary claim.
- The model excels when it can analyze both source code and the running program (“live + source”), but its performance drops when only static source is available.
- Mythos shows strong judgment in rejecting false positives, yet it can be overly literal, discarding true positives that lack formal evidence.
- In reverse‑engineering and native‑code vulnerability discovery, Mythos demonstrates substantial strength and can triage its own findings as well as those of competitor models.
- Visual‑acuity tests reveal the model is practically effective at selecting correct browser actions, though it is not pixel‑precise for exact coordinates.
- Running Mythos is expensive—estimated at ~5× the cost of an Opus model—but when normalized by cost, it remains competitive for high‑accuracy web‑vulnerability hunting, outperforming Opus 4.6 yet falling short of GPT‑5.5.
- Overall, Mythos is a powerful tool for source‑code audits and exploit validation, but teams must weigh its precision demands and resource costs against alternative models.
Introduction to Mythos and Its Claims
Anthropic’s Mythos AI model debuted in early April with a bold promise: to uncover considerably more software vulnerabilities than any existing AI system. The claim quickly attracted attention from security researchers and offensive‑security firms eager to test the limits of generative AI in vulnerability discovery. XBOW, an autonomous offensive‑security outfit, launched its own AI‑driven testing arsenal against Mythos Preview to verify whether the model lives up to the hype. Their independent evaluation aimed not only to confirm vulnerability‑yield numbers but also to probe ancillary capabilities such as judgment, reverse engineering, native‑code analysis, and visual interaction with live web interfaces. The results, released shortly after the tests, provide a nuanced picture of where Mythos shines and where it falls short.
XBOW’s Independent Evaluation Framework
XBOW structured its assessment around several distinct test suites, each targeting a specific facet of Mythos’ purported abilities. First, they measured raw vulnerability yield by feeding the model source code alone and then the same code operating in a live environment. Second, they evaluated judgment by presenting ambiguous snippets and recording the model’s tendency to accept or reject potential findings. Third, reverse‑engineering exercises challenged Mythos to dissect binaries, firmware, and embedded‑system code. Fourth, visual‑acuity tests required the model to manipulate a browser interface to locate and click UI elements. Finally, XBOW factored in operational cost estimates supplied by Anthropic to calculate efficiency metrics. This multi‑dimensional approach allowed the researchers to isolate strengths and weaknesses rather than relying on a single aggregate score.
Vulnerability Detection: Source Code vs Live Environments
The cornerstone of XBOW’s validation confirmed Anthropic’s headline claim: Mythos Preview uncovers a markedly higher number of vulnerabilities than preceding models. However, the source of that advantage is contextual. When granted access to both the source code and the running program (live + source), Mythos consistently outperforms baselines, spotting flaws that arise only during execution—such as race conditions, improper privilege handling, or environment‑specific misconfigurations. In contrast, when limited to static source code alone, the model’s detection rate drops noticeably, suggesting that many of its findings depend on runtime behavior or dynamic data flow that static analysis cannot capture. This nuance does not diminish Mythos’ power; rather, it highlights the importance of pairing the model with execution‑aware testing or instrumentation for maximal coverage.
Judgment and False Positive/Negative Trade‑offs
Mythos exhibits a discerning eye when it comes to filtering out noise. In judgment‑focused tests, the model rejected false positives at a rate superior to its predecessors, indicating a conservative bias that helps reduce alert fatigue for analysts. Yet this conservatism has a downside: when evidence does not strictly satisfy Mythos’ formal criteria—such as incomplete call‑stack information or ambiguous data‑flow paths—the model sometimes discards true positives. Users have reported that the model requires precise, well‑structured prompts to achieve optimal recall; vague or overly broad queries can lead it to overlook exploitable flaws. Consequently, security teams must invest in prompt engineering or wrapper scripts that supply the necessary contextual clues to balance Mythos’ precision with sufficient recall.
Reverse Engineering and Native‑Code Analysis
Beyond web‑oriented vulnerabilities, Mythos shows considerable strength in reverse‑engineering and native‑code contexts. XBOW’s tests tasked the model with triaging its own vulnerability reports as well as those generated by competing AI systems. Mythos consistently demonstrated the ability to prioritize findings, correlate related issues, and explain why certain binaries behave unexpectedly. In firmware and embedded‑system scenarios—where resources are constrained and instruction sets vary—the model reasoned through unusual calling conventions and memory‑mapped I/O to uncover hidden weaknesses. This capability suggests that Mythos can serve as a force multiplier for specialized audits, particularly when combined with traditional disassembly tools that provide the raw binary data the model consumes.
Visual Acuity and Browser Interaction Tests
To gauge Mythos’ applicability to dynamic web testing, XBOW conducted visual‑acuity exercises that required the model to interact with a live browser interface. The model was asked to locate specific UI elements—buttons, links, form fields—and execute clicks or inputs as part of an automated attack chain. While Mythos was not pixel‑perfect when queried for exact screen coordinates, it proved practically effective at selecting the correct actions based on semantic cues (e.g., identifying a “Submit” button by its label and surrounding context). This level of performance enables the model to drive basic interaction scripts for vulnerability scanners, though fine‑grained tasks such as canvas‑based drawing or precise drag‑and‑drop may still require supplemental heuristics or human oversight.
Cost Considerations and Efficiency Normalization
Anthropic has disclosed that Mythos Preview will be roughly five times more expensive to run than an Opus model, a figure that raises concerns about scalability for large‑scale continuous‑integration pipelines. XBOW addressed this by normalizing performance metrics against estimated operational cost. When adjusting for expense, Mythos Preview is not dramatically inefficient; it delivers high accuracy that justifies its price for missions where missing a vulnerability carries substantial risk. However, on a pure cost‑adjusted benchmark, the model does not rank as the absolute best‑in‑class—other models such as GPT‑5.5 achieve comparable or superior vulnerability yields per unit cost. Organizations must therefore weigh the incremental security gain against budget constraints, potentially reserving Mythos for high‑value targets or periodic deep dives rather than constant low‑latency scanning.
Comparative Performance Against Opus and GPT Models
In head‑to‑head trials with a fixed token budget for web‑vulnerability discovery, Mythos Preview outperforms Opus 4.6, confirming its advantage within Anthropic’s own product line. Yet when pitted against GPT‑5.5 under the same constraints, Mythos falls short, indicating that the latter’s architecture or training data yields a higher vulnerability‑detection efficiency in this specific domain. These results do not invalidate Mythos’ core strength; rather, they situate it within a competitive landscape where multiple models excel under different conditions. Teams seeking the highest raw detection count might favor GPT‑5.5, while those prioritizing tight integration with Anthropic’s tooling or requiring specialized reverse‑engineering insight may still opt for Mythos despite its higher price tag.
Overall Assessment and Implications for Security Teams
XBOW’s comprehensive testing leads to a clear set of takeaways for practitioners. Mythos Preview is unequivocally stronger at uncovering software vulnerabilities than earlier AI models, especially when it can analyze both source and runtime behavior. Its judgment leans toward precision, reducing false positives but occasionally sacrificing recall unless prompts are meticulously crafted. In reverse‑engineering and native‑code scenarios, the model demonstrates robust reasoning abilities that can augment traditional binary analysis tools. Visual interaction with browsers is competent for high‑level tasks, though not suited for ultra‑fine‑grained UI manipulation. Cost‑wise, Mythos is a premium offering; when normalized by expense, it remains a viable option for high‑assurance audits but is not the most economical choice for bulk, low‑risk scanning. Consequently, security leaders should consider Mythos as a specialized asset—ideal for critical codebases, complex firmware, or situations where deep contextual understanding outweighs pure volume of alerts.
Future Outlook and Recommendations
Looking ahead, the evolving interplay between model size, training data, and inference cost will likely reshape the value proposition of systems like Mythos. Anthropic may introduce optimized variants or tiered pricing that make the model more accessible for continuous security testing. In the interim, organizations can maximize Mythos’ return on investment by coupling it with complementary tools: static analyzers for broad coverage, dynamic instrumentation to supply the live‑plus‑source context Mythos craves, and human experts to validate edge cases the model might overlook. Prompt‑library development and automated wrapper scripts can also mitigate the model’s sensitivity to query phrasing. By strategically deploying Mythos where its strengths align with organizational risk profiles, security teams can harness its advanced reasoning capabilities while keeping budget and operational overhead under control.

