Modern LLM Pentesting Goes Far Beyond Prompt Injection

LLM pentesting is a controlled security assessment of AI solutions based on large language models that goes beyond prompt injection. Specialists test whether an attacker can gain access to confidential data, influence the model’s information sources, misuse tools and APIs, bypass authorization, or trigger a real business action. Successfully manipulating a model’s response alone does not constitute a confirmed risk - the risk is determined by the data, functions, and actions that the manipulation provides access to.

An LLM system can work with corporate data, RAG, APIs, and external tools, so a potential attack can go far beyond manipulating its responses. In our practice, we often see businesses limit the security assessment of an AI solution to prompt injection testing and consider this sufficient proof that it is secure. Let’s look at what else is tested during LLM pentesting and what risks these tests can help identify.

Why the attack surface has changed

The attack surface is the sum of all potential paths through which an attacker can attempt to gain access to a system, its data, or functions, or otherwise affect how it operates. The more components and integrations a system has, the broader its attack surface becomes.

Modern LLM solutions are not limited to a model that simply receives a request and generates a response. They can use RAG to work with corporate knowledge bases, store context in memory, access APIs and external tools, while AI agents can also perform certain actions in other systems.

As a result, potential attack targets include not only the model itself and its prompts, but also the data, context sources, access mechanisms, integrations, and functions available to the LLM. For example, weak access controls in RAG can create a risk of confidential documents being exposed, while excessive permissions granted to an AI agent can allow it to perform an action that the user themselves is not authorized to perform.

In our practice, the most dangerous findings have often emerged precisely at the “interfaces” between system components rather than within the model itself.

This is why the approach to security testing has also changed: the more capabilities an LLM solution has and the more deeply it is integrated into business systems, the more potential attack paths need to be considered during penetration testing.

What is LLM pentesting and why is it conducted?

LLM pentesting (LLM penetration testing) is a controlled security assessment of systems and applications that use large language models (LLMs). During the test, cybersecurity specialists simulate real-world attack scenarios and look for vulnerabilities. Depending on the architecture of a specific product, the assessment may cover not only the model itself but also the data connected to it, RAG, APIs, external tools, authorization mechanisms, and other components.

The main goal of this type of penetration testing is to understand whether an attacker can exploit weaknesses in LLM solutions to have a real impact on the system or business. For example, an attacker might gain access to confidential data they are not authorized to access, manipulate the information used by the model, bypass established restrictions, or initiate an unauthorized action through an AI agent.

Important: during LLM pentesting, we assess not only what the model “says” in its response, but what the system is actually capable of doing - whether it is possible to obtain data, invoke a function, or complete an operation.

Unlike hackers, pentesters reproduce such scenarios within controlled and pre-agreed boundaries. As a result, the company receives not simply a list of potential AI risks, but identified vulnerabilities, an assessment of their potential impact, and remediation recommendations to prevent real attackers from exploiting these weaknesses.

In our view, one of the best-known ways to influence LLM behavior is prompt injection. Let’s look at what this technique involves and what role it plays in LLM pentesting.

Prompt Injection in LLM Pentesting

Prompt injection as a real attack

Prompt injection is an attack technique in which an attacker attempts to use specially crafted instructions to alter an LLM’s behavior or make it ignore established restrictions. Malicious instructions can be submitted directly in a user request or reach the model unnoticed through documents, websites, and other files it reads.

The intended outcome of an attack is shaped by the capabilities available within the system. An attacker may try to obtain information that the model should not disclose, alter the logic of its response, or prompt an AI agent to misuse an available tool or function. Therefore, the danger of prompt injection is determined not only by whether the attacker succeeds in influencing the model, but also by what data and functions the model has access to and what it is capable of doing within the system.

Prompt injection in LLM pentesting

During LLM penetration testing, specialists reproduce prompt injection scenarios, both direct and indirect, in a controlled manner to determine whether the model can be manipulated and what real-world consequences this could have for a specific product. In this way, testing helps identify weaknesses in security mechanisms and understand the potential impact of an attack on data, functions, and business processes.

At the same time, resistance to prompt injection is only one area of LLM security testing. A comprehensive pentest should also take into account other risks related to the architecture and capabilities of the specific product.

What LLM pentesting covers beyond prompt injection

When testing for prompt injection, pentesters check whether specially crafted or hidden instructions can be used to alter an LLM’s behavior. But for businesses, it is important to know more than just whether the model can be “tricked.” The entire LLM system needs to be tested: whether an attacker can access confidential data, influence the model’s information sources, use its tools and APIs, bypass access restrictions, or cause a dangerous action to be performed.

Depending on the architecture of the specific LLM solution, such testing may cover the following areas:

1. Sensitive information disclosure

An LLM solution may work with personal data, internal documents, financial information, commercial data, and other information that should not be accessible to all users.

OWASP specifically identifies Sensitive Information Disclosure as one of the key risks for LLM applications. PII, proprietary information, and sensitive business data are among the types of information that may be exposed. Prompt injection can be one way to trigger such a leak, but OWASP explicitly notes that restrictions can be bypassed through both prompt injection and other methods.

What is tested: whether it is possible to obtain information through the LLM that a specific user should not be able to see, and whether the system reliably separates public data from confidential data.

2. RAG and knowledge base security

Relevant for solutions that use RAG/vector stores.

RAG gives an LLM access to relevant information stored in external sources, such as corporate knowledge bases and documents, before it formulates a response. This expands the system’s capabilities but also creates an additional attack surface.

OWASP lists Vector and Embedding Weaknesses as a distinct category of security risks. Potential consequences include unauthorized access, data leakage, cross-context information leaks, and data poisoning. This is particularly important in multi-tenant environments, where a single vector database may contain information belonging to different groups of users.

A common issue we encounter is that teams restrict document access in the web interface but do not carry the same access controls over to retrieval. For example, an employee may not have direct access to a confidential document, but if the AI assistant uses a shared knowledge base and the retrieval process does not take the user’s permissions into account, the assistant could potentially retrieve information from that document and include it in its response.

What is tested:

• Whether users can gain access through the LLM to documents or pieces of data they would otherwise be unauthorized to access.
• Whether data belonging to different users and tenants is properly separated.
• Whether malicious or manipulative content in the knowledge base can be used to influence the system’s responses.

3. Data and context manipulation

An attacker does not necessarily have to interact directly with the LLM: in certain architectures, they may attempt to influence the information that the model will later receive as context.

For RAG systems, OWASP explicitly describes data poisoning: malicious or compromised data can enter the knowledge base and subsequently influence the model’s responses. Sources may include unverified or deliberately inserted content.

What is tested:

• Whether information that the system considers trusted can be injected, replaced, or modified.
• Whether context sources are verified.
• Whether such content can alter the LLM’s subsequent behavior.

4. Misuse of tools, functions, and APIs

Relevant if the LLM or agent can perform actions rather than just generate text.

An LLM or AI agent may have access to external functions and systems: searching for information, modifying records, sending messages, creating requests, working with files, CRM systems, payment services, or other services.

In OWASP terminology, this is closely related to the risk of Excessive Agency: it arises when an LLM system has excessive functionality, permissions, or autonomy, allowing compromised outputs to lead to dangerous actions. OWASP also specifically emphasizes that modern agentic systems introduce risks such as tool misuse and agent privilege escalation.

For a regular chatbot, a dangerous outcome may be an incorrect response or the disclosure of confidential information. For an AI agent, the consequence may be a real action performed in another system.

What is tested is whether the system can be made to:

• Invoke a function that the user should not be able to control.
• Pass dangerous parameters.
• Perform an action without the required confirmation.
• Use excessive capabilities granted to the agent.

5. Authorization and privilege abuse

The level of access granted to an LLM or AI agent may not be the same as the user’s. A risk arises if a low-privilege user can use AI to take advantage of the broader permissions granted to the agent itself or an associated service account.

For example: user → AI agent → API → corporate system. The user’s access rights do not allow them to modify a specific record. But if the agent accesses the API with broader permissions and the backend does not verify the end user’s permissions, AI can become a way to bypass established restrictions.

For modern agentic systems, OWASP separately identifies Identity & Privilege Abuse risks, while Excessive Agency also directly links dangerous consequences to excessive permissions.

What is tested:

• Whether the user’s permissions are correctly enforced when the LLM interacts with other systems.
• Whether the agent has excessive privileges.
• Whether AI can be used to access other users’ resources or perform a prohibited operation.

A comprehensive LLM pentest is not a narrow prompt injection assessment. It should take the actual architecture of the LLM product into account and assess what attack opportunities are created by the model’s interaction with data, RAG, tools, APIs, the authorization system, and other components.

What these areas of LLM pentesting mean for businesses

A successful prompt injection is not the end of the assessment, but a reason to take it further. The particular value of LLM pentesting lies in how far the attack path can be traced within a specific system.

The five areas described above should not be viewed as a mandatory set of tests - the scope is determined by the architecture of the specific product. For example, RAG testing is unnecessary for a system that does not use RAG, while tools/function calling testing is unnecessary if the model cannot invoke such functions:

• If the system uses RAG, testing covers retrieval, vector stores, data access, and potential data poisoning.
• If the system uses tools/APIs, testing covers potential misuse of functions and unsafe actions.
• If there is an AI agent, permissions, autonomy, and authorization become particularly important.
• If the system works with confidential data, scenarios involving unauthorized disclosure of that data are tested.

Expertise and controlled LLM pentesting

Simulating real-world attacks on LLM solutions requires an understanding not only of how the language model works, but also of the product architecture, integrations, access mechanisms, and business logic. An experienced specialist selects attacks relevant to the system, tests them safely, and demonstrates the business risks posed by the vulnerabilities identified.

Unlike the actions of a real attacker, LLM pentesting is conducted in a controlled manner: the scope, permitted methods, and testing boundaries are agreed upon in advance. 

This makes it possible to test potentially dangerous scenarios while minimizing the risks of disrupting the product’s operation, damaging data, or unintentionally affecting business processes.

As part of penetration testing, our team:

• Defines the scope of the assessment based on the architecture and capabilities of the specific LLM solution.
• Simulates relevant attack scenarios targeting the model and its related components.
• Tests the risks of unauthorized access to data, functions, and integrated systems.
• Assesses the real-world consequences that identified vulnerabilities could lead to.
• Provides a report with prioritized risks and recommendations for remediation.

Conclusion

Prompt injection is an important but far from the only area of security testing for LLM solutions. Modern LLM pentesting takes the architecture of the specific product into account and may cover the protection of confidential data, RAG security, resilience to context manipulation, the use of tools and APIs, authorization, and other relevant risks.

Do you know how secure your LLM solution is beyond prompt injection? The Datami team is ready to simulate attack scenarios relevant to your product, determine the real-world impact of identified vulnerabilities, and provide recommendations for remediation.