Researchers at AI safety startup Zenity demonstrated how a number of extensively used enterprise AI assistants may be abused by risk actors to steal or manipulate information.
The Zenity researchers showcased their findings on Wednesday on the Black Hat convention. They shared a number of examples of how AI assistants may be leveraged — in some instances with none consumer interplay — to do the attacker’s bidding.
Enterprise instruments are more and more built-in with generative AI to spice up productiveness, however this additionally opens cybersecurity holes that may very well be extremely beneficial to risk actors.
As an illustration, safety specialists demonstrated up to now how the combination between Google’s Gemini gen-AI and Google Workspace productiveness instruments may be abused by immediate injection assaults for phishing.
Researchers at Zenity confirmed final yr how they might hijack Microsoft Copilot for M365 by planting specifically crafted directions in emails, Groups messages or calendar invitations that the attacker assumed would get processed by the chatbot.
This yr, Zenity’s specialists disclosed related assault strategies concentrating on ChatGPT, Copilot, Cursor, Gemini, and Salesforce Einstein.
Within the case of ChatGPT, the researchers focused its integration with Google Drive, which permits customers to question and analyze recordsdata saved on Drive. The assault concerned sharing a specifically crafted file — one containing hidden directions for ChatGPT — with the focused consumer (this requires solely realizing the sufferer’s electronic mail handle).
When the AI assistant was instructed by the sufferer to course of the malicious file, the attacker’s directions could be executed, with none interplay from the sufferer. Zenity demonstrated the dangers by getting ChatGPT to go looking the sufferer’s Google Drive for API keys and exfiltrate them. Commercial. Scroll to proceed studying.
Within the case of Copilot Studio brokers that have interaction with the web — over 3,000 situations have been discovered — the researchers confirmed how an agent may very well be hijacked to exfiltrate info that’s obtainable to it. Copilot Studio is utilized by some organizations for customer support, and Zenity confirmed how it may be abused to acquire an organization’s total CRM.
When Cursor is built-in with Jira MCP, an attacker can create malicious Jira tickets that instruct the AI agent to reap credentials and ship them to the attacker. That is harmful within the case of electronic mail programs that robotically open Jira tickets — lots of of such situations have been discovered by Zenity.
In an illustration concentrating on Salesforce’s Einstein, the attacker can goal situations with case-to-case automations — once more lots of of situations have been discovered. The risk actor can create malicious instances on the focused Salesforce occasion that hijack Einstein when they’re processed by it. The researchers confirmed how an attacker might replace the e-mail addresses for all instances, successfully rerouting buyer communication by a server they management.
In a Gemini assault demo, the specialists confirmed how immediate injection may be leveraged to get the gen-AI software to show incorrect info. In Zenity’s instance, the attacker received Gemini to supply a checking account owned by the attacker when the sufferer requested a sure buyer’s account.
The ChatGPT and Copilot Studio weaknesses have been patched, however the remainder have been flagged as ‘received’t repair’ by distributors.
Associated: Vibe Coding: When Everybody’s a Developer, Who Secures the Code?
Associated: AI Guardrails Beneath Hearth: Cisco’s Jailbreak Demo Exposes AI Weak Factors
Associated: Google Gemini Tricked Into Displaying Phishing Message Hidden in E mail
