Anthropic’s Mythos AI has been in the spotlight since its debut in April, lauded for its exceptional ability to uncover software vulnerabilities. However, its performance in other domains remains complex and varied.
Exceptional Vulnerability Detection
Mythos has gained attention for its unparalleled skill in identifying software vulnerabilities compared to other AI models. XBOW, a leading autonomous offensive security firm, has thoroughly evaluated Mythos Preview to validate these claims. They confirm that Mythos Preview significantly surpasses existing models in this specific capability.
Gary McGraw’s insights from two decades ago emphasize the importance of understanding both code and design to spot defects. XBOW’s assessment of Mythos shows that the model excels when evaluating live systems in conjunction with source code, though it is less effective when limited to source code alone.
Capabilities Beyond Code Analysis
Mythos’s proficiency extends to native code vulnerability discovery and reverse engineering. XBOW’s tests reveal that Mythos can effectively analyze and triage results, offering insights into complex firmware and embedded systems.
In judgment, Mythos has shown improved accuracy in rejecting false positives compared to earlier AI models. However, it occasionally overlooks true positives when the criteria are not met precisely, indicating a need for exact prompts to achieve optimal results.
Cost and Efficiency Considerations
While Mythos is a powerful tool, its operational costs are notable. Anthropic has indicated that Mythos will be five times more expensive than the Opus model, prompting discussions about its cost-effectiveness. XBOW suggests that a less expensive model, with more runtime, might achieve similar accuracy at a lower expense.
The conclusion drawn by XBOW is that while Mythos is not inefficient, it is not the top performer in their benchmarks. It demonstrates superior performance compared to Opus 4.6 for web vulnerabilities within a fixed token budget, yet it is surpassed by GPT5.5.
Overall, Mythos stands out for its ability to audit source code and discover vulnerabilities, although its performance in validating exploits is less robust. The model’s judgment can sometimes be too literal and conservative, exaggerating the practical impact of its discoveries.
Despite these challenges, XBOW concludes that Mythos Preview is a formidable tool for identifying potential vulnerabilities, showcasing remarkable capabilities across various technical tasks.
