What AI Penetration Testing Includes: A Breakdown of 12 Key Security Checks

AI penetration testing is a security assessment of a system in which AI is the target of an attack: testers determine whether it is possible to alter the model’s behavior, gain access to other users’ data, or force an agent to perform an unwanted action. The testing scope is defined individually based on the architecture, integrations, and business logic of each solution. Accordingly, the checks performed as part of an AI penetration test may vary, but 12 main areas can be identified.

We are often asked whether we perform the same testing for all AI solutions and whether there is a standard list of tests for AI Penetration Testing. No, there is no universal checklist: the set of checks depends on the type of solution, architecture, functions, data, integrations, level of autonomy, and potential risks.

In this article, we will cover 12 key types of checks that we perform during penetration testing of AI systems. This is not a fixed or exhaustive standard: some checks may not apply during penetration testing.

A Comprehensive AI Pentest Starts with the Real Attack Surface

AI pentesting (AI Penetration Testing) is practical security testing of an AI system: a controlled simulation of an attack that assesses whether an attacker can alter the system’s behavior, gain unauthorized access to data, or force it to perform a dangerous or unauthorized action.

The scope of testing is determined by the attack surface – the set of components, interactions, and access points of an AI system through which an attacker can influence its behavior, data, or actions:

• chat interface and system prompt;
• RAG sources;
• connected tools and APIs;
• agent permission management interfaces;
• persistent memory;
• external content processing modules.

The broader the attack surface, the more vectors need to be tested – test scenarios depend on the system’s actual capabilities, not just on the type of model.

We divided the 12 key areas of security testing into four groups: the model, data and context, agent actions, and the system environment itself.

We want to note from the outset that we apply all these checks in practice, but we do not automatically include them in every pentest just for the sake of a formal checklist. For each specific AI system, we select only the relevant checks based on its architecture, functions, and attack surface – without unnecessary tests that have no practical value in a particular case.

Model Behavior Protection

Our team starts an AI pentest by testing the model’s behavior – this is the first thing an attacker tries to bypass before gaining access to the system’s data or tools. Let’s look at three main types of such checks:

Security Check 1. Direct Prompt Injection and Resistance to Jailbreak Attacks

What is tested: whether the model can be forced to ignore system rules, change its role, disclose restricted information, or perform a prohibited action. Testing covers different phrasings, obfuscation, translations, and multi-step delivery of the same instruction.

Risk: a successful prompt injection can force the model to execute commands that violate the product’s rules.

Business impact: unsafe responses, violations of internal policies, regulatory scrutiny, reputational damage, or escalation to attacks on connected systems.

Example: a user gradually changes the context of the conversation, after which the assistant provides information it was supposed to keep hidden.

Expected evidence: a reproducible jailbreak scenario, the rule that was violated, the conditions for a successful bypass, and the actual impact on the product.

Security Check 2. Indirect Prompt Injection

What is tested: whether the AI follows malicious instructions hidden in a document, email, web page, RAG source, or response from an external tool.

Risk: indirect injection turns data that the model is only supposed to analyze into a command.

Business impact: an attacker influences the AI without direct access to it, alters its outputs, triggers integrations, or gains access to corporate data.

Example: the AI receives a document containing a hidden prompt injection and, instead of analyzing it, triggers an action in the CRM.

Expected evidence: the complete path from external content to a change in model behavior, a tool call, or data leakage, as well as confirmation that the content was not checked before being added to the context.

Security Check 3. Guardrail Resilience and Policy Compliance

What is tested: whether guardrails – the system’s protective mechanisms – remain resilient to jailbreak scenarios after translation, paraphrasing, splitting a request into parts, prolonged dialogue, or the use of external context. The entire control chain is tested, not just the input filter.

Risk: the system blocks obvious unsafe requests but allows the same intent when expressed in a different form.

Business impact: the company relies on controls that work only in limited scenarios and do not cover the entire workflow.

Example: a request is blocked in English but gets through after translation or after being split across several messages.

Expected evidence: guardrail bypass scenarios, the system component where the control failed, and the actual consequence.

Jailbreaks, prompt injection, and weak guardrails become critical not on their own, but when they open a path to data, tools, or business operations.

Data and Context Protection

An AI system may work with confidential data, knowledge bases, RAG, and memory, so it is important to test not only access to this information but also the ability to influence the context the model receives.

Security Check 4. Disclosure of Confidential Data and Internal Instructions

What is tested: whether system instructions, credentials, personal information, internal documents, other users’ conversations, or hidden context can be obtained – not only through direct requests but also gradually, by collecting individual fragments.

Risk: the model discloses information that should not be accessible to a specific user.

Business impact: confidentiality breaches, fines, leakage of commercial information, compromised credentials, and loss of customer trust.

Example: an internal assistant returns fragments of another client’s contract through a series of follow-up requests.

Expected evidence: the type and source of the data, the user’s role, the scale of the leak, and possible consequences. Disclosure of the system prompt is not always critical: what matters is what data it contains and whether it helps bypass other controls.

Security Check 5. RAG Authorization and Client Environment Isolation

What is tested: whether a user can obtain documents or fragments through AI that they would not normally have access to.

Risk: the search and authorization logic returns relevant documents or fragments but does not apply the permissions of the specific user, their role, or the boundaries of the client environment.

Business impact: data leakage between departments, clients, or organizations, as well as violations of contractual and regulatory requirements.

Example: an employee asks the assistant to compare projects, and data from a restricted client account appears in the response.

Expected evidence: confirmation of which restricted data was obtained and why the system allowed access. The model should not independently determine which documents a user is allowed to access.

Testing should confirm that authorization is applied to search, retrieval, citations, and subsequent agent actions, not only to the chatbot interface.

Security Check 6. Knowledge Base, Context, and Memory Poisoning

What is tested: whether an attacker can add information that changes future AI responses, decisions, or actions – in RAG documents, long-term memory, summaries of previous conversations, or shared agent context.

Risk: malicious or false content enters the knowledge base, context, or memory and affects subsequent sessions.

Business impact: systematically incorrect decisions, dissemination of false information, errors in customer support, or internal processes.

Example: an attacker gains the ability to add or modify a document in a source that the system considers trusted. After indexing, the AI uses a false financial instruction in its responses to users.

Expected evidence: the duration of the impact, affected users or processes, and the ability to completely remove the poisoned context. A poisoned document may first alter the context, then activate a tool, and ultimately cause data leakage.

Security of AI Agent Actions

AI agents can not only generate responses but also interact with tools and external systems, performing real actions on behalf of the user. Therefore, during a pentest, we check whether an attacker can exploit these capabilities to perform unauthorized or dangerous actions.

Security Check 7. Tool and Function Call Abuse

What is tested: whether the AI can call an unintended tool, pass dangerous or unverified parameters, repeat an operation, or build a dangerous chain of permitted functions.

Risk: the model uses a legitimate integration for an unintended purpose.

Business impact: modification or deletion of data, unwanted messages, erroneous transactions, and disruption of internal processes.

Example: the agent is only supposed to read an email but instead creates a record in the CRM or sends a reply on behalf of an employee.

Expected evidence: the actual tool call, the action performed, the permissions used, and the reason why the system allowed the operation.

Security Check 8. Identification, Authorization, and the Principle of Least Privilege

What is tested: whether the AI performs actions with the user’s actual permissions after verifying their role, or uses an overly privileged system account.

Risk: a user gains more capabilities through the agent than they have in the standard interface.

Business impact: privilege escalation, access to other users’ accounts, modification of critical data, and violation of client environment isolation.

Example: an employee without editing permissions asks the AI to update a record, and the agent performs the operation using its own privileged account.

Expected evidence: the action, the user’s role, the permissions used, and the difference between direct access and access through AI. The system prompt is not an authorization mechanism.

Security Check 9. Goal Hijacking and Human Oversight

What is tested: whether an agent’s goal can be changed, a mandatory workflow step can be skipped, approval can be bypassed, or a malicious task can be passed to another agent.

Risk: the AI uses permitted functions but pursues a different business objective.

Business impact: unauthorized decisions, bypassing internal approvals, erroneous payments, or actions performed on behalf of the company.

Example: a malicious instruction in an incoming task or external document changes the agent’s goal: the interface asks the user to approve an update to one record, but the actual tool call requests a bulk update to multiple records.

Expected evidence: the complete workflow, the bypassed control, the actual action, and the information the user saw during approval.

A professional test examines not an individual tool, but the entire path from instruction to the executed business action.

Protecting the AI System’s Operational Environment

The security of an AI system depends not only on the model, data, and agent actions, but also on how the system processes outputs, uses resources, and responds to suspicious activity.

Security Check 10. Improper Handling of Model Outputs

What is tested: whether external components treat the model’s output as trusted data or commands. These may include web interfaces, APIs, automated code execution systems, or other agents.

Risk: the next component in the chain treats malicious model output as a command and executes it.

Business impact: data modification, application compromise, execution of a dangerous operation, or propagation of the attack to other components.

Example: the AI generates a structured response that an internal system uses as a command without additional validation.

Expected evidence: the path from the input data to the subsequent action and the component that failed to validate the model’s output.

Security Check 11. Resource Exhaustion and Agent Loops

What is tested: whether the AI can be forced to make an excessive number of calls, repeat tasks, initiate long-running workflows, or use expensive tools.

Risk: legitimate functions become a means of exhausting resources.

Business impact: a sharp increase in AI costs, service unavailability, delays in critical processes, and overload of external systems.

Example: the agent cannot complete a task and repeatedly calls the model and tools without limits.

Expected evidence: the number of operations, duration, resources consumed, and the control that should have stopped the process.

Security Check 12. Detection, Logging, and Containment

What is tested: whether the security team can detect, investigate, and stop an attack after it begins. The visibility of prompts, tool calls, agent decisions, and executed operations is assessed, as well as the ability to quickly disable a dangerous component.

Logs should not create a new channel for data leakage: access to them, masking of confidential data, and retention periods should be controlled separately.

Risk: an attack occurs without sufficient logs, alerts, or ways to limit its consequences.

Business impact: longer response times, a greater scale of data leakage, more difficult investigations, and an inability to confirm which data or users were affected.

Example: the agent performs a series of unusual operations, but the security team sees only the final response in the chat.

Expected evidence: whether an alert was generated, what data was recorded in the logs, whether the attack chain can be reconstructed, and whether execution can be stopped quickly. Monitoring does not prevent attacks – it helps detect and contain them in time.

These are the areas that the Datami team checks during AI Penetration Testing. However, not every product needs all twelve at once: a chatbot without external systems does not require tool-calling tests, while a simple RAG system does not require goal hijacking testing.

How AI Penetration Testing Works

AI pentesting includes 7 key penetration testing stages, each of which we adapt to the specifics of the AI system.

Within these 7 stages, we perform the 12 AI system security checks described above.

What Businesses Get from AI Penetration Testing

Based on the pentesting results, businesses receive not just a list of vulnerabilities, but tangible benefits:

• A real picture of risks instead of hypotheses: visibility into which vulnerabilities exist in the model, data, agent actions, or operational environment and what negative consequences they may lead to.
• Validated attack chains: an understanding of how individual weaknesses, such as a vulnerable guardrail and excessive agent permissions, combine into a real attack scenario rather than remaining isolated findings.
• A report that is clear to both the team and management: developers receive step-by-step recommendations for addressing security gaps in AI systems, while the business receives an assessment of the impact on the product and customers.
• A proactive approach instead of reacting to an incident: testing an AI solution identifies vulnerabilities in advance – before they are discovered by someone else: an attacker, customer, or regulator.

Conclusions

Individual vulnerabilities may not be critical, but the risk increases when several weaknesses form a complete attack chain: bypassing security barriers opens the way to tool calls, which in turn can lead to changes in CRM data.

Therefore, the result of a high-quality AI pentest is not simply a list of testing findings, but an understanding of which security weaknesses exist, which attacks may succeed, what business consequences they may lead to, and which controls should be strengthened first.

The 12 security checks are not a universal checklist that is mechanically applied to every system, but the main areas of testing within an AI pentest. Their scope, depth, and combination depend on the architecture of your AI system, data, integrations, access permissions, and level of autonomy.

Does your company use an AI solution and want to make sure it is secure? Test it against real-world attack scenarios with the Datami team.