Email systems often include AI Assistants for users, which can be exploited by attackers as a form of Living off the Land (LotL) once they compromise an account. This tactic becomes especially concerning as it provides attackers with a versatile tool without the need for external resources.
Gaining control of an email account is the most challenging aspect of this method, yet attackers persistently overcome this hurdle. Once they have access, they can utilize the built-in AI Assistant of the compromised account, bypassing traditional security measures.
Exploring the Potential for Abuse
Researchers at Barracuda Networks have examined how attackers might misuse AI chatbots in email systems. In a controlled, simulated environment, they demonstrated how an attacker could escalate privileges from a lower-level user to a CEO, without raising alarms.
The experiment focused on using the AI to maintain stealth and gather information. An attacker can manipulate the chatbot to remove traces of their activities, such as instructing it to create rules to hide specific emails. Gathering organizational insights through the chatbot further aids in crafting believable phishing emails.
Phishing with Contextual Precision
With a compromised account, attackers can craft phishing emails that bypass security filters by appearing as legitimate internal communications. By mimicking the writing style of the compromised user, these emails can deceive even vigilant recipients.
In the simulation, researchers directed the chatbot to draft an email for a budget approval process. This email included a link that, when clicked by the CEO, led to a session token hijack, allowing attackers to access the CEO’s account and circumvent multifactor authentication (MFA).
Implications and Future Outlook
Following the initial breach, attackers can continue to exploit the AI Assistant to maintain access and further compromise sensitive information. In the simulated attack, researchers illustrated how an attacker could redirect a substantial financial transaction by impersonating the CEO.
This research underscores the potential for AI chatbots to be misused in cybersecurity breaches. While the experiment was controlled, it highlights the significant threat posed by AI technologies if not properly secured and monitored.
The findings emphasize the need for robust security measures to prevent such exploitations. As AI becomes increasingly integrated into digital platforms, understanding and mitigating these risks is crucial for maintaining security and trust in digital communications.
