Researchers from Stanford College, Carnegie Mellon College, and Grey Swan AI have unveiled ARTEMIS, a classy AI agent framework that demonstrates exceptional aggressive capabilities in opposition to seasoned cybersecurity professionals.
Within the first-ever complete comparability of AI brokers in opposition to human consultants in a stay enterprise atmosphere, ARTEMIS positioned second total, outperforming 9 of ten skilled penetration testers whereas sustaining considerably decrease operational prices.
The groundbreaking examine evaluated each the AI agent and ten extremely certified human cybersecurity professionals on an in depth college community comprising roughly 8,000 hosts throughout 12 subnets.
The ARTEMIS framework recognized 9 legitimate vulnerabilities with a powerful 82% valid-submission price, demonstrating technical sophistication similar to that of the strongest human contributors.
The analysis, revealed in December 2025, represents a important shift in understanding AI’s precise capabilities in real-world cybersecurity operations.
ARTEMIS AI and Human Penetration Testers
In contrast to present cybersecurity AI brokers that depend on inflexible single-agent architectures, ARTEMIS employs an modern multi-agent framework that includes dynamic immediate era, limitless sub-agents, and computerized vulnerability triaging.
The system consists of three core parts: a supervisor managing the workflow, a swarm of specialised sub-agents, and a classy triage module designed for vulnerability verification and classification.
The framework addresses elementary limitations in present agent scaffolds by enabling prolonged operational horizons via clever session administration, context summarization, and resumable workflows.
ARTEMIS multi-agent framework
ARTEMIS achieved peak parallelism with eight concurrent sub-agents, demonstrating efficiencies inconceivable for human operators working sequentially.
Present frameworks reminiscent of Codex and CyAgent, when evaluated on the identical goal atmosphere, considerably underperformed relative to most human contributors, highlighting the important significance of correct architectural design.
Past technical capabilities, ARTEMIS demonstrated compelling financial benefits. Probably the most environment friendly ARTEMIS variant (A1) operated for $18.21 per hour, roughly equal to $37,876 annualized at customary 40-hour workweeks.
This represents a dramatic price discount in comparison with the typical U.S. penetration tester, who earns roughly $125,034 yearly. The extra refined A2 configuration prices $59 per hour whereas attaining comparable vulnerability discovery charges, nonetheless considerably cheaper than human professionals.
This financial benefit carries profound implications for enterprise safety posture. Steady penetration testing, traditionally impractical on account of skilled labor prices, turns into economically viable via AI brokers like ARTEMIS.
Organizations can now conduct ongoing safety assessments at a fraction of conventional engagement prices whereas sustaining the technical depth mandatory for significant vulnerability discovery.
The analysis reveals necessary limitations that inform the event trajectory of AI-enabled cybersecurity instruments. ARTEMIS reveals increased false-positive charges in comparison with human contributors, notably when parsing ambiguous HTTP responses and authentication flows that people readily interpret via graphical interfaces.
Evaluating AI Brokers to Cybersecurity Professionals
The framework struggles with GUI-based interactions, lacking the important TinyPilot distant code execution vulnerability that 80% of human contributors efficiently recognized. This limitation displays broader constraints in present massive language mannequin capabilities.
Conversely, ARTEMIS demonstrated distinctive strengths unavailable to human operators. Its command-line interface proficiency enabled the profitable exploitation of legacy techniques that fashionable browsers refuse to load.
The agent efficiently exploited an outdated IDRAC server utilizing SSL certificates bypass methods whereas people deserted the goal on account of browser failures.
Performed below complete IRB approval with strict security protocols, the examine maintained safety all through the evaluation. Actual-time monitoring prevented out-of-scope conduct, and collaborative coordination with college IT employees ensured accountable vulnerability disclosure and patching.
The researchers’ resolution to open-source ARTEMIS displays their conviction that improved defensive instruments serve broader cybersecurity pursuits.
The ARTEMIS examine offers important proof for knowledgeable regulatory decision-making relating to AI’s offensive capabilities. With risk actors more and more leveraging AI in cyber operations, a complete real-world analysis of AI capabilities permits defenders to develop simpler countermeasures.
The analysis demonstrates that whereas AI brokers can not but match essentially the most skilled professionals, they current a transformative functionality that calls for severe safety consideration and proactive defensive funding.
Comply with us on Google Information, LinkedIn, and X for every day cybersecurity updates. Contact us to function your tales.
