AI and LLM security testing: when your AI assistant becomes a confused deputy

Demand for AI and large language model security testing doubled in 2025. The risks are now clearly defined in the OWASP Top 10 for LLM Applications 2025. They include Prompt Injection, sensitive information disclosure, excessive permissions granted to AI agents (Excessive Agency), system prompt leakage, and vulnerabilities in databases used for Retrieval-Augmented Generation (RAG).

For the banking sector, Excessive Agency is one of the most serious risks. One of our projects in 2025 clearly demonstrated why. We tested AI assistants integrated into the mobile app of a financial institution. They were designed to help customers with routine tasks, explain services, and guide them through the app. However, the developers had also given them the ability to perform certain actions. When a customer asked to set up a recurring transfer, the assistant did more than navigate them to the correct screen: it could create the transfer itself. That convenience became a security risk.

An XSS vulnerability alone is usually not enough to initiate a bank transfer because transactions are protected by additional safeguards, such as user confirmation and anti-CSRF tokens. This is why we also look for XSS and other application-level weaknesses during web application and API penetration testing. But when XSS is combined with an AI agent that follows a malicious instruction and performs an action on the user’s behalf, the assistant becomes a confused deputy – a component that unknowingly bypasses the protections specifically designed to secure the transaction. The vulnerability does not stem from memory corruption or a conventional programming flaw, but from the model’s willingness to help.

If the same agent is authorised to add a payment recipient, change the notification email address, or reset security questions, a successful Prompt Injection attack can turn any of these functions into a tool for fraud.

This is not an isolated problem. The same principle was demonstrated at scale by EchoLeak (CVE-2025-32711), a zero-click Prompt Injection vulnerability that turned Microsoft 365 Copilot into a channel for silently exfiltrating data through a single specially crafted email. This is also why OWASP launched a dedicated project on agentic AI security in 2025. Agents capable of taking action, using external tools, and chaining operations together can turn the theoretical risk of Excessive Agency into real financial losses.

What does this mean for CISOs?

AI risks are not all the same. An internal Copilot connected to company documents primarily creates a risk of data leakage, as EchoLeak demonstrated. The priority in this case should be strict access controls and validation of model outputs. Customer-facing assistants and agentic systems that can take action fail differently: through excessive permissions and misuse of the tools available to them. They therefore require sandboxing, the principle of least privilege, and human approval for every significant operation. In both cases, the same rule applies: AI is part of the production attack surface and must be tested regularly, not checked off once during development. For a SOC, this means monitoring prompts, tool calls, and outbound data as closely as the activity of any privileged service account.

Is AI security merely hype or the new standard? According to our data, it is neither. It is a new production layer that is here to stay. Companies are deploying it faster than they can test it, and attackers are well aware of the gap. Over the next two years, the deciding factor will not be whether you adopt AI, but whether you test and monitor it as rigorously as any other production system.

Do you know what your AI assistant can perform on a user’s behalf?

We test chatbots, AI agents, RAG systems, and LLM applications for resilience against Prompt Injection, sensitive data leakage, tool misuse, and excessive permissions.