XBOW tests Anthropic Mythos Preview for faster vulnerabilities
Anthropic
TLDR: XBOW’s security team ran Mythos Preview through benchmarks, interactive tests, and live exploit validation, finding major gains in source code vulnerability hunting, plus mixed judgment and weaker exploit validation. The model cost remains high at about 5x Opus, prompting questions about cheaper accuracy routes.
Key Takeaways:
- XBOW used a standard agent test system plus new angles like threat modeling, live access, and native app bug discovery.
- Mythos Preview cut false negatives 42% on XBOW web exploit benchmarks, and 55% when given site source code.
- Source code reads drive wins, but live exploit proof needs validation harnesses and multiple tools, because judgment can be literal and overly conservative.
Mythos Preview looks less like a magic hacker and more like a sharper code reader with a bias toward formal evidence. The real headline is that getting from lead to working exploit still takes the full XBOW body, not just the brain 🛡️.
Mythos Preview looks less like a magic hacker and more like a sharper code reader with a bias toward formal evidence. The real headline is that getting from lead to working exploit still takes the full XBOW body, not just the brain 🛡️.
Q&A
If Mythos Preview is so strong at finding leads in code, why does exploit validation still lag?
Because exploitability depends on deployment details like dependencies, configuration, and component interactions. XBOWs harness with live site actions is what turns reasoning into working evidence.
What tradeoff does the 42% or 55% false negative drop imply for a security team triaging alerts?
You may see fewer missed candidates, but you still need validation and context. Mythos can narrow the search, yet it cannot replace live testing and exploit proof.
Why did Mythos Preview struggle more when live site access was removed, even for benchmarks where the bug is in code?
XBOW says live-site access still helps models reconstruct the attacker view. Even when the vulnerability originates in code, real exploit paths often rely on runtime behavior and boundaries.
How should teams interpret Mythos Preview being literal and conservative in safety and command safety checks?
It can reduce some false alarms but also discard true leads that do not match strict rule language. Better prompts, explicit threat models, and post filtering matter.
Could organizations save money by routing tasks between Mythos Preview and cheaper models?
XBOWs cost framing suggests yes. Often the best workflow is letting a cheaper model run more attempts, while Mythos Preview handles the parts where source code reasoning yields outsized gains.
No comments yet. Be the first to share your thoughts!