Modern LLM Pentesting Goes Far Beyond Prompt Injection

Modern LLM pentesting checks not only whether the model's behavior can be changed through prompt injection. It shows what data, tools, and business operations an attacker could gain access to if such manipulation succeeds. For an AI assistant connected to RAG, CRM, corporate APIs, or payment functions, this path is what determines the real risk.

Getting the model to ignore an instruction is not the same as gaining access to a restricted document or carrying out an operation. Therefore, a high-quality pentest does not stop at the response in the chat. It traces what the system allows an attacker to do next.

Why the attack surface has changed

Early mass-market LLM applications were often limited to a chat interface. In modern products, the model is often part of a broader system: it retrieves documents through RAG, uses memory, calls APIs, works with tools, and delegates tasks to other agents.

In practice, some of the most dangerous findings arise not within the model itself, but at the boundaries between components. Imagine a situation where RAG returns a document that the system trusts without verification, the model generates a tool call, and the backend executes it with excessive privileges. Each component may appear secure on its own, but their interaction creates a complete attack path.

The key takeaway: a pentest limited to the chat window simply does not see most real-world risks.

In simplified form, this path can be shown as follows:

User or External Content → LLM → RAG / Memory → Tools / API → Business Action

Each transition between components should be treated as a potential trust boundary – especially when the data source, authorization context, or security boundary changes. It is at these transitions that typical problems arise:

• context is passed without verification;
• the model gains access to data it was not authorized to access;
• authorization is checked incompletely;
• a tool call is formed incorrectly;
• an operation with real business impact is executed without additional control.

The attack surface expands every time the model is connected to a new data source, permission, tool, or business process. This means that the scope of testing cannot be determined in advance – it depends on how many such connections exist in a particular product and how critical the actions they enable are.

Before diving into what exactly should be tested, it is worth distinguishing between three related concepts that are often conflated:

1. Model evaluation measures the model's capabilities, quality, reliability, or behavior according to defined scenarios and metrics.
2. AI red teaming tests how the system behaves under deliberate adversarial influence and may cover safety, misuse, and other risk categories.
3. LLM pentesting looks for practically exploitable weaknesses in a specific deployed product, its integrations, authorization mechanisms, and business operations.

In the NIST ARIA program (NIST AI 700-2 report), the evaluation of AI applications is divided into model testing, red teaming, and field testing. This is not a classification of pentesting types, but it clearly demonstrates why model evaluation does not replace testing the system in the context of its real-world use.

Prompt injection – an entry point, not the entire attack

Prompt injection describes only one stage of an attack: an attempt to influence the model's instructions, context, or behavior. It can be direct – through a user message – or indirect, when a malicious instruction enters the context from a document, webpage, email, memory, or response from a third-party tool. In an agentic system, such influence can propagate further between components and agents.

The mere fact that the model can be influenced does not yet demonstrate the actual risk. The danger is determined by what happens next – what actions the system allows a model whose behavior has already been manipulated to perform.

Consider the same influence in three different architectures:

1. In a public chatbot without access to internal systems, the consequences are more likely to remain at the level of content, accessible context, reputational damage, or resource abuse.
2. In a corporate assistant with RAG, the same influence can lead to the unauthorized disclosure of a restricted document.
3. In an agent with access to APIs, it can lead to data modification or the execution of an operation.

The attack pattern in all three cases can be practically identical. The difference lies entirely in what the AI system allows the model to do next.

Therefore, the main question during a pentest is not “Was it possible to change the model's response?” but “What confirmed outcome does this make it possible to achieve in the specific system?”

What modern LLM pentesting should actually cover

There is no single “template” or universal testing scope for LLM pentesting. Testing is always built around the architecture of a specific product: its data flows, access rights, integrations, and business functions.

After all, not every AI has RAG, long-term memory, MCP, or complex multi-agent scenarios – so there is no point in testing all products in the same way. Below are three levels that clearly explain why simply testing the chat window is insufficient for a complete assessment here.

Data layer

A team may correctly restrict a user's access to documents in the web interface but fail to apply the same rules during retrieval. As a result, the user cannot see the file directly but may receive fragments of it through the AI assistant's response.

At this level, testing should cover:

• access controls in RAG and vector stores;
• direct and indirect sources of context;
• user and tenant isolation;
• writing data to session and long-term memory;
• storage of secrets and personal data;
• poisoning of memory and retrieval content.

A useful question to ask yourself: if a user does not have permission to open a document directly, can the model still find it, quote it, or use it to generate a response?

The main risk is simple: untrusted data influences the model's behavior, while errors in retrieval or memory expose information that the user should not be able to see. This can result in confidential data leakage, tenant isolation violations, exposure of personal data, or long-term influence on future sessions through “poisoned” memory.

Execution layer

Tools turn the model from a text generator into a component capable of changing the state of the system. It can create a record in a CRM, send a message, place an order, or initiate a payment – depending on the functions available to it.

Imagine a scenario where a third-party MCP server returns not only data but also an instruction that influences the model's next decision. A pentest should verify whether the system will treat such output as trusted and whether it can trigger another tool.

Next, testing should determine:

• whether the model can call a function it should not have access to, including a hidden or undocumented one;
• whether the input and output data of a tool call are validated and whether parameters can be substituted;
• whether critical or irreversible actions can be triggered without human confirmation;
• whether there are restrictions on what exactly the tool is allowed to do.

The backend must recheck authorization, parameter validation, and the permissibility of the operation. An LLM's decision cannot be sufficient grounds for executing a critical action.

The main risk: if function calling or an MCP integration is insufficiently restricted and the backend trusts the LLM's decision instead of performing additional verification, a manipulated model can trigger the same operations as a legitimate request – including initiating a payment, modifying a record, or performing another operation that the user never requested.

Authorization layer

One of the most common and at the same time most dangerous architectural mistakes is giving an agent more permissions than the user on whose behalf it operates. In this case, AI becomes a way to bypass the existing authorization model.

It is essential to verify:

• whether the user's identity is preserved throughout the entire chain;
• whether each downstream system independently checks permissions;
• whether different users share an overly privileged service account;
• whether an agent can delegate an unavailable operation to another agent;
• whether separate confirmation is required for financial, irreversible, or external actions.

When this boundary disappears, local manipulation of the model turns into an action performed on behalf of another user, gaining administrative access, or spreading an attack between agents – in other words, into an impact that extends far beyond a single user or a single session.

Logging and auditability span all three layers: if an action at any of these levels cannot be traced after the fact, it is difficult to assess the actual scale of the incident or explain its significance to the CTO, CISO, or Product Owner of the AI solution.

The purpose of this division is not to test every theoretically possible AI component, but to identify the ones the product actually uses and trace all the paths through which they can provide access to data, permissions, tools, and business operations.

Depending on the architecture, testing may also cover the AI supply chain, third-party models and libraries, system prompts, model endpoints, key management, processing of generated data, rate limits, resource constraints, and deployment infrastructure.

There is no universal set here: the testing scope includes the components that the product actually uses and through which confidentiality, integrity, availability, or the cost of operations can be affected.

Test what the system can do, not just what the model says

We often see teams record an unusual model response and consider the testing complete. But this is only the beginning of the path – it is important to shift the focus from “dangerous text in the response” to a measurable technical or business outcome.

There is a significant difference between the following states: the model agreed to execute a prohibited instruction; the system actually disclosed restricted data; a tool call was generated; the backend accepted and executed that call; the operation actually changed the state of the system; the action affected a user, payment, order, document, or business process.

A high-quality pentest does not end at the first point on this list – it is important not to stop there, but to trace the attack path to its final impact, to the point where it is possible to clearly state what exactly happened in the system, rather than just what words the model produced.

For example, the model agreed to “issue a refund” – this is not yet a finding with confirmed business impact. It is necessary to determine whether a tool call was generated, whether the backend accepted it, whether authorization was successful, and whether the status of a real order changed. Only then can the impact be assessed and the controls that failed be identified.

For each identified risk, it is worth documenting:

• the entry point;
• the source of untrusted context;
• the path taken between components;
• the required role or access;
• the control that was bypassed;
• the action achieved;
• the data, users, or processes put at risk;
• evidence of impact;
• the recommended owner of the remediation.

This is the format that turns a technical finding into an argument that is understandable to both the security team and the business.

Prompt injection demonstration vs. full LLM pentest

The difference between a superficial demonstration and a full-fledged assessment is easy to see when placed side by side:

A successful prompt injection is a reason to continue testing, not the point at which it should end.

Questions worth asking before choosing an LLM pentesting provider

1. How do you define the pentest scope – based on a fixed checklist or the product architecture? A good answer shows that the provider analyzes the architecture, data sources, data flows, available tools and integrations, roles and permissions, and business-critical actions – and adapts the testing to the specific system rather than offering one standard package for any AI product.
2. Do you test the entire application or only the model interface – and does the report separate findings by layer? It is important to determine whether the testing includes the backend, RAG, memory, tools, APIs, and downstream systems, and whether the report separates findings at the model, application, and infrastructure levels – this helps properly distribute vulnerability remediation among teams.
3. Do you test authenticated roles and permission boundaries? A single public-user scenario does not reveal the risks associated with different roles, tenants, and service accounts.
4. Do you assess function calling, tools, and MCP integrations? This is critical for systems capable of performing real actions rather than simply responding with text.
5. Do you test indirect inputs? The provider should account for documents, web content, emails, tool outputs, and other sources of context, not just user messages.
6. How do you confirm real impact while controlling testing risk? The provider should explain how it distinguishes an unwanted response from an exploitable vulnerability, how it safely confirms the impact without damaging production data, and how it links findings to a specific business risk.
7. Do you cover multi-step and multi-agent scenarios? This is important for products with orchestration, task delegation, and autonomous actions.
8. Do you test the AI supply chain and infrastructure if they are included in the agreed scope? This may include third-party models, libraries, model endpoints, access keys, logs, rate limits, and the deployment environment.

LLM Pentest Scope Checklist

Data

• Are both direct and indirect input sources taken into account?
• Are system instructions protected from being overwritten through context?
• Does RAG follow the same access permissions as the main interface?
• Are vector stores isolated between users and tenants?
• Can one user access data from another user's session?
• Can malicious content be written to long-term memory?
• Are secrets or personal data stored in context or memory?

Execution

• Can the model call a function it should not have access to?
• Is what each tool is allowed to do restricted?
• Are the instructions, data, and permissions of third-party MCP servers verified?
• Are the parameters and results of API calls validated?
• Can tool call parameters be substituted?
• Are operations that cannot be reversed restricted?
• Do critical and irreversible actions require independent confirmation?

Access and Authorization

• Does the agent operate within the permissions of the specific user on whose behalf it is acting?
• Is the user's identity preserved at every step of the chain?
• Do different users share an overly privileged service account?
• Does the agent have only the permissions required for the specific task?
• Can one tenant affect another tenant's data or actions?
• Does each downstream system independently verify permissions?
• Can an agent delegate an operation to another agent that it cannot perform itself?

Business Logic

• Can a manipulated model initiate or modify a financial transaction?
• Can the model modify a record without independent verification of its permission to do so?
• Can an approval step be bypassed through model manipulation?
• Can the model send a message on behalf of the company without oversight?
• Is every operation confirmed independently of the LLM's decision?
• Does the next system verify the model's output before using it?
• Is it possible to trace after the fact what exactly happened in the system?

This checklist is a guide, not a set of instructions to follow blindly. There is no need to test what you do not have. Simply filter out what is irrelevant and make sure you have accounted for every real vector: from connected databases and access permissions to integrations and specific actions within the system.

Conclusion

Prompt injection remains an important part of LLM pentesting, but it primarily demonstrates the ability to influence the model's behavior. Such a result alone does not yet demonstrate what real risk to the system or business the manipulation may lead to.

The level of risk is determined by what data the model has access to, what permissions the agent receives, which systems it interacts with, and what operations it can perform. This is why the scope of an LLM pentest should not be based on a universal template – it should reflect the architecture, integrations, permissions, and business functions of the specific product.

If, after testing, you only know that the model can be made to violate an instruction, the assessment is not yet complete. You need to understand what data, permissions, tools, and operations this influence leads to – and which controls should stop it at each step. It is the ability to reproduce the product's real architecture and safely confirm the achievable impact that distinguishes a full LLM pentest from a prompt injection demonstration.

Want to check whether your current security assessment accounts for all the real capabilities of your AI product?