What Does AI Penetration Testing Include? A Practical Breakdown of 12 Critical Security Checks

AI penetration testing is a security assessment in which the AI itself is the target: testers determine whether they can change the model's behavior, access someone else's data, or force an agent to perform an unwanted action. Two companies may order the same service – AI penetration testing – but receive fundamentally different testing scopes and results: one team may only check whether chatbot instructions can be bypassed, while another may test the model, RAG, memory, tools, agent permissions, and complete attack chains leading to real business consequences.

Two AI Penetration Tests Can Deliver Completely Different Results

This article focuses primarily on LLM applications: chatbots, RAG systems, copilot solutions, and AI agents. Testing traditional ML models involves different scenarios – for example, testing for model evasion, extraction, or inference – and is not covered in this breakdown.

Before ordering the service, it is important to clearly distinguish between two areas that are often confused:

1. AI Penetration Testing – testing in which the AI system is the target of the attack.
2. AI-powered Penetration Testing – using AI as a tool to identify traditional vulnerabilities in conventional applications.

AI red teaming is a related adversarial testing methodology that can be part of comprehensive AI Penetration Testing, but is not synonymous with it.

Red teaming usually focuses on identifying unwanted model behavior, while a comprehensive pentest also checks authorization, control over tool calls, access to data, and business consequences at the level of connected systems.

In 2026, the scope of AI Penetration Testing has expanded significantly. AI systems no longer just generate text:

• agents gain access to CRM systems, email, databases, and internal APIs;
• MCP servers and tools allow the model to interact with external systems;
• long-term memory influences behavior in future sessions;
• AI independently plans and executes business actions;
• multi-agent workflows pass tasks between multiple components.

The more autonomy, data, and integrations an AI system has, the broader the testing scope should be.

A comprehensive test starts with the real attack surface

A universal set of prompts is not enough. Before testing, the team determines:

• What exactly the AI can do.
• What data it has access to.
• Which external systems and tools are used.
• Whether RAG and memory are present.
• What user roles exist in the system.
• Which actions the AI agent performs independently and where human confirmation is required.
• Which business processes could be affected by an attack.

An AI that only answers questions is simply a reference tool. Real business value is created by agents that take action: modifying data in a CRM, sending emails, processing payments, or executing code. And AI should be tested through the lens of its actual actions and access, not simply by the type of model.

Let's break down 12 key security tests across four areas: the model, data and context, agent actions, and the system environment itself.

Protecting model behavior

Security test 1. Direct prompt injection and resistance to jailbreak attacks

What is tested: whether the model can be forced to ignore system rules, change its role, disclose restricted information, or perform a prohibited action. Testing covers different phrasings, obfuscation, translations, and multi-step delivery of the same instruction.

Risk: a successful prompt injection causes the model to execute commands that violate the product's rules.

Business impact: dangerous responses, violations of internal policies, regulatory issues, reputational losses, or escalation to attacks on connected systems.

Example: a user gradually changes the context of the conversation, after which the assistant provides information it was supposed to keep hidden.

Expected evidence: a reproducible jailbreak scenario, the rule that was violated, the conditions under which the bypass succeeded, and the actual impact on the product.

Security test 2. Indirect prompt injection

What is tested: whether the AI follows malicious instructions hidden in a document, email, webpage, RAG source, or response from an external tool.

Risk: indirect injection turns data that the model is only supposed to analyze into a command.

Business impact: an attacker influences the AI without direct access to it, changes its outputs, triggers integrations, or gains access to corporate data.

Example: the AI receives a document containing a hidden prompt injection and, instead of analyzing it, triggers an action in the CRM.

Expected evidence: the complete path from external content to a change in model behavior, a tool call, or data leakage, as well as confirmation that the content was not verified before being added to the context.

Security test 3. Guardrail resilience and policy compliance

What is tested: whether guardrails – the system's protective mechanisms – remain resilient to jailbreak scenarios after translation, rephrasing, splitting a request into parts, prolonged dialogue, or the use of external context. The entire control chain is tested, not just the input filter.

Risk: the system blocks obvious dangerous requests but allows the same intent when presented in a different form.

Business impact: the company relies on a control that works only in limited scenarios and does not cover the entire workflow.

Example: a request is blocked in English but goes through after translation or after being split across several messages.

Expected evidence: guardrail bypass scenarios, the system component where the control failed, and the actual consequence.

Jailbreaks, prompt injection, and weak guardrails become critical not on their own, but when they open a path to data, tools, or business operations.

Protecting data and context

Security test 4. Disclosure of confidential data and internal instructions

What is tested: whether it is possible to obtain system instructions, credentials, personal information, internal documents, other users' conversations, or hidden context – not only through a direct request, but also gradually by collecting separate fragments.

Risk: the model discloses information that should not be available to a specific user.

Business impact: confidentiality breaches, fines, leakage of commercial information, compromised credentials, and loss of customer trust.

Example: an internal assistant returns fragments of another client's contract through a series of follow-up requests.

Expected evidence: the type and source of the data, the user's role, the scale of the leak, and the possible consequences. Disclosure of the system prompt is not always critical: what matters is what data it contains and whether it helps bypass other control mechanisms.

Security test 5. RAG Authorization and Tenant Isolation

What is tested: whether a user can obtain documents or fragments through AI that they would not normally have access to.

Risk: retrieval and authorization logic returns relevant documents or fragments but does not apply the specific user's permissions, role, or client environment boundaries.

Business impact: data leakage between departments, clients, or organizations, as well as violations of contractual and regulatory requirements.

Example: an employee asks the assistant to compare projects, and data from a restricted client account appears in the response.

Expected evidence: confirmation of which restricted data was obtained and why the system allowed access. The model should not independently determine which documents the user is allowed to access.

Testing should confirm that authorization is applied to search, retrieval, citations, and subsequent agent actions, not just to the chatbot interface.

Security test 6. Knowledge base, context, and memory poisoning

What is tested: whether an attacker can add information that changes future AI responses, decisions, or actions – in RAG documents, long-term memory, summaries of previous conversations, or shared agent context.

Risk: malicious or false content is stored in memory and influences subsequent sessions.

Business impact: systematically incorrect decisions, the spread of unreliable information, errors in customer support, or internal processes.

Example: an attacker gains the ability to add or modify a document in a source that the system considers trusted. After indexing, the AI uses a false financial instruction in its responses to users.

Expected evidence: the duration of the impact, the users or processes affected, and the ability to completely clear the poisoned context. A poisoned document may first alter the context, then activate a tool, and ultimately cause a data leak.

Security of AI agent actions

Security test 7. Abuse of tool and function calls

What is tested: whether the AI can call an unintended tool, pass dangerous or unverified parameters, repeat an operation, or build a dangerous chain from permitted functions.

Risk: the model uses a legitimate integration for an unintended purpose.

Business impact: modification or deletion of data, unwanted messages, erroneous transactions, and disruption of internal processes.

Example: the agent is only supposed to read an email but instead creates a record in the CRM or sends a response on behalf of an employee.

Expected evidence: the actual tool call, the action performed, the permissions used, and the reason the system allowed the operation.

Security test 8. Identity, authorization, and the principle of least privilege

What is tested: whether the AI performs actions using the user's actual permissions after verifying their role or whether it uses an overly privileged system account.

Risk: the user gains more capabilities through the agent than they have in the standard interface.

Business impact: privilege escalation, access to other users' accounts, modification of critical data, and tenant isolation violations.

Example: an employee without editing permissions asks the AI to update a record, and the agent performs the operation through its own privileged account.

Expected evidence: the action, the user's role, the permissions used, and the difference between direct access and access through AI. A system prompt is not an authorization mechanism.

Security test 9. Goal hijacking and human oversight

What is tested: whether it is possible to change the agent's goal, skip a mandatory workflow step, bypass approval, or pass a malicious task to another agent.

Risk: the AI formally uses permitted functions but pursues a different business objective.

Business impact: unauthorized decisions, bypassing internal approvals, erroneous payments, or actions performed on behalf of the company.

Example: a malicious instruction in an incoming task or external document changes the agent's goal: the interface asks the user to approve an update to one record, but the actual tool call contains a bulk modification of several records.

Expected evidence: the complete workflow, the control that was skipped, the actual action, and the information the user saw during approval.

A professional test examines not an individual tool, but the entire path from the instruction to the executed business action.

Protecting Operations and the Surrounding System

Security test 10. Unsafe handling of model outputs

What is tested: whether external components treat the model's output as trusted data or commands. These may include web interfaces, APIs, automated code execution systems, or other agents.

Risk: the next component in the chain treats malicious model output as a command and executes it.

Business impact: data modification, application compromise, execution of a dangerous operation, or propagation of the attack to other components.

Example: the AI generates a structured response that an internal system uses as a command without additional verification.

Expected evidence: the path from input data to the subsequent action and the component that failed to verify the model's output.

Security test 11. Resource exhaustion and agent loops

What is tested: whether the AI can be forced to generate an excessive number of calls, repeat tasks, launch long-running workflows, or use expensive tools.

Risk: legitimate functions become a means of exhausting resources.

Business impact: a sharp increase in AI costs, service unavailability, delays in critical processes, and overload of external systems.

Example: the agent cannot complete a task and repeatedly calls the model and tools without limits.

Expected evidence: the number of operations, duration, resources consumed, and the control that should have stopped the process.

Security test 12. Detection, logging, and containment

What is tested: whether the security team can detect, investigate, and stop an attack after it begins. The visibility of prompts, tool calls, agent decisions, and executed operations is assessed, as well as the ability to quickly disable a dangerous component.

Logs should not create a new channel for data leakage: access to them, masking of confidential data, and retention periods should be controlled separately.

Risk: the attack occurs without sufficient logs, alerts, or ways to limit its impact.

Business impact: longer response times, a greater scale of data leakage, more difficult investigation, and an inability to confirm which data or users were affected.

Example: the agent performs a series of unusual operations, but the security team sees only the final response in the chat.

Expected evidence: whether an alert was generated, what data was recorded in the logs, whether the attack chain can be reconstructed, and whether execution can be stopped quickly. Monitoring does not prevent attacks – it helps detect and contain them in time.

How AI penetration testing works in practice

A typical AI penetration testing process begins with the following steps, though the depth of testing and specific scenarios depend on the system's architecture, data, integrations, and level of autonomy.

1. Scope and architecture review – the team identifies components, data, roles, tools, and critical business functions.
2. Threat modeling and attack chain design – realistic scenarios are developed based on the system's capabilities.
3. Manual and automated adversarial testing – automation helps expand coverage, but manual testing is necessary to assess business logic and complex attack chains.
4. Impact verification – the team safely verifies whether a vulnerability leads to an actual data leak, operation, workflow change, or financial loss.
5. Reporting and remediation recommendations – evidence, impact description, root cause, and recommendations are provided for each identified issue.
6. Retesting – after fixes are implemented, the primary scenario and alternative ways to bypass guardrails are tested again.

A high-quality finding should include: the affected component, prerequisites, reproduction steps, evidence, business impact, severity level, root cause, remediation recommendations, and retesting status. A screenshot of an unwanted response without an attack chain and confirmed business impact is an insufficient result for a report.

What AI penetration testing is not

Comprehensive AI penetration testing is not limited to:

• running hundreds or thousands of jailbreak prompts;
• automated scanning without manual verification;
• testing only the chatbot interface;
• mechanically going through the OWASP Top 10 for LLM Applications 2025 without adapting scenarios to the specific architecture;
• searching for unwanted responses without verifying business impact;
• AI-powered scanning for traditional web vulnerabilities;
• AI red teaming without assessing application logic, authorization, and integrations.

Red teaming can be part of the methodology, but comprehensive AI penetration testing should also provide reproducible results, confirmed business impact, and practical remediation recommendations.

How to evaluate the quality of the service and report

Before choosing a provider, it is worth asking the following questions about the testing scope:

1. Is the entire AI system tested, not just the chatbot?
2. Are RAG, memory, tools, and agent actions taken into account?
3. Are different roles and access levels tested?
4. Are direct and indirect prompt injection and jailbreak scenarios tested?
5. Are real actions in connected systems tested?
6. Does the team build complete attack chains?
7. Is the business impact of each identified issue explained?
8. Is reproducible evidence provided?
9. Is the testing scope adapted to the specific architecture?
10. Is retesting included in the cost of the service?

Signs of a high-quality service:

• the testing scope depends on the system's business functions;
• manual and automated checks are combined with validation of each result;
• guardrail resilience against bypasses is tested;
• identified issues demonstrate real impact;
• the model, data, agent actions, and integrations are assessed separately;
• the report contains clear recommendations;
• retesting is included.

Warning signs:

• the main metric is the number of prompts executed;
• all clients are offered the same checklist;
• RAG, tools, and permissions are not included in the testing scope;
• the report contains only unwanted model responses;
• real-world consequences are not tested;
• reproduction steps are missing;
• business impact is not explained;
• fixes are not retested.

Conclusions

The completeness of AI penetration testing is determined not by the number of prompts or the length of the vulnerability list, but by how deeply the system's actual architecture is tested. A high-quality test should answer three questions:

1. How can an attacker change the behavior of the AI system?
2. What data, tools, or business operations can this provide access to?
3. Can the company prevent, detect, and contain the real consequences of an attack?

Before an AI pentest, it is important to understand the basics: the solution's architecture, access permissions, integrations, and levels of autonomy. Only then will you get testing aimed at the real threats specific to your system rather than a formal “template-based” assessment.