AI-driven customer service bots are increasingly entrusted with critical tasks within companies, including managing customer profiles, billing inquiries, and account modifications. However, recent research reveals that these bots are susceptible to manipulation, enabling attackers to extract sensitive data or impersonate legitimate users.
Phishing Risks in Chatbot Transcripts
One significant vulnerability involves the transcript features in many support chatbots. Attackers can exploit these by inserting malicious text into a conversation, then using the transcript function to send phishing emails that seem to originate from a trusted support address. Such tactics can increase the credibility of phishing attempts, especially when emails appear to come from a legitimate domain like [email protected].
Email spoofing further exacerbates the problem, as some AI systems rely on the visible From header for user identification, ignoring other sender verification fields. This flaw allows attackers to use an authenticated but misleading address, tricking the AI into associating the email with a victim’s account.
Exploiting AI for Unauthorized Access
In another attack scenario, an AI bot might process a request that appears to originate from a genuine customer. By manipulating email fields, an attacker could receive sensitive data such as billing information. This vulnerability stems from weak authentication checks and prompt injection tactics, as highlighted by security researcher Inti De Ceukelaire at DEF CON 34.
The risk extends to systems requiring one-time passcodes for sensitive operations. Attackers can bypass rate limits by altering email formats, exploiting inconsistencies in email normalization across different systems, potentially gaining unauthorized access.
Mitigating AI Security Vulnerabilities
AI bots linked to support inboxes pose a threat of exposing third-party account codes. Attackers might instruct the bot to act on malicious commands, leading to the release of sensitive data like password reset codes. These incidents are challenging to prevent, as they often bypass human checks through techniques like multipart messages or hidden HTML cues.
Additionally, knowledge-base poisoning is a growing issue. AI systems using retrieval-augmented generation can be misled by maliciously indexed information, resulting in the dissemination of false instructions or codes. To counter these risks, companies must separate untrusted content from system instructions, enforce stringent identity verification, and limit AI bot permissions.
Organizations should treat AI systems as privileged automation tools rather than simple conversational interfaces. Implementing robust security measures will help protect valuable data and maintain customer trust in AI-assisted services.
