In a recent evaluation conducted by the OWASP GenAI Security Project, ChatGPT demonstrated its ability to assist a user with limited security expertise in compromising an administrator account. This controlled hacking test highlighted the potential of AI to reduce the technical barrier for executing certain cyber attacks, while still necessitating human involvement.
AI’s Role in Simplifying Security Breaches
During the test, the participant relied heavily on ChatGPT to execute commands, interpret results, and receive guidance for actions like changing passwords and accessing accounts. This collaboration illustrates how AI can aid in lowering the skill threshold required for cyber attacks, although it does not eliminate the need for human decision-making.
The insights were published in the LLM Exploit Generation Phase 2 report, which compared human-guided chat interactions and autonomous agents against simulated targets provided by Hack The Box. These environments were strictly for training and did not involve real-world breaches. Researchers focused on factors such as success rates, human input, cost implications, safety protocols, and potential failures.
Details of the Controlled Test
The primary scenario involved a machine named Silentium, featuring a concealed staging website running a vulnerable version of Flowise. ChatGPT generated a testing guide that enabled the operator to follow procedures using tools like nmap and curl. The AI facilitated the discovery of the staging site, navigating the login process, and eventually changing the administrator’s password.
Despite the AI’s assistance, the task was not entirely devoid of skill. The test assumed basic computer literacy, command-line competency, and the ability to rephrase queries when necessary. Setting up a Kali environment was excluded from the test parameters.
Challenges and Observations
Researchers noted an incident where a response vanished due to an error, but subsequent requests succeeded. This anomaly could not be reproduced or conclusively tied to a vulnerability in OpenAI’s protections. The autonomous trials, which involved various configurations including CAI and HexStrike, also faced challenges. While they managed to locate staging sites and identify vulnerabilities, they could not complete the final takeover step due to safety mechanisms halting progress.
OWASP emphasizes the importance of human oversight and recommends implementing approval protocols before executing significant actions. The report advocates caution against overestimating AI’s capabilities based on benchmark scores alone, emphasizing that while AI can accelerate attack phases, reliable execution still relies on human expertise.
For security teams considering AI deployment, integrating human verification processes is crucial to mitigate risks and enhance incident response strategies.
