AI Penetration Testing: What to Test Beyond Prompt Injection

A CISO-focused guide to testing AI applications as complete systems: identity, APIs, retrieval, prompts, tools, agents, output handling and the permissions behind every action.

AI-enabled applications inherit the normal web, API, cloud and identity attack surface, then add a model that can interpret untrusted language, retrieve internal data and, in some systems, call tools. That changes the question a penetration test needs to answer.

A system prompt leak may be interesting. Cross-tenant document retrieval, unauthorized tool execution or an agent sending data to an external service is materially different. Security leaders should test the whole path from attacker-controlled input to data access or action.

Why AI security is an application security problem

Start with the architecture, not a prompt list

The right test plan depends on what the application can reach. A customer-support chatbot that only answers from public documentation has a different risk profile from an internal assistant connected to SharePoint, email and ticketing, and both differ again from an agent that can create users, change cloud resources or issue refunds.

Before testing, map where untrusted content enters, where identity is established, how prompts are assembled, what retrieval layer is queried, which tools can be called, where model output is rendered or executed, and which systems trust the result. That map usually reveals more useful test cases than a long list of jailbreak strings.

The model is one component inside an application. Authentication, API authorization, cloud permissions, retrieval filters, prompt assembly, memory, third-party connectors and agent tools still determine the real blast radius. A good AI assessment therefore combines conventional application testing with adversarial model and workflow testing.

This is also why toy jailbreaks make poor risk evidence. If a model produces disallowed text but cannot reach sensitive data or privileged functions, the business impact may be limited. If a harmless-looking document can manipulate a retrieval workflow and cause an agent to send internal data externally, the impact is much clearer.

What should be in scope

Three deployment patterns create three different tests

• Conversational assistant: focus on data exposure, session isolation, system-prompt assumptions, output handling and abuse of application APIs.
• RAG application: add document ACLs, tenant isolation, ingestion, poisoning, metadata filters, deletion, embedding/vector access and indirect prompt injection.
• Agentic application: add tool authorization, delegated identity, approval gates, memory, inter-agent trust, MCP or other tool protocols, egress and side effects.

A mature engagement may cover all three because production products often combine them. The important point is that “LLM testing” is not one fixed test shape.

Prompt injection: test the consequence, not the phrase

Direct prompt injection comes from the user. Indirect prompt injection arrives through content the system retrieves or reads, such as a web page, document, email or ticket. In both cases, the tester should trace what happens after the model follows the instruction.

Useful tests ask whether the injected instruction can bypass a data boundary, influence a tool call, change a destination, suppress an approval step or expose hidden information. Record the model version, system configuration and exact path to impact so the result can be reproduced as closely as the system permits.

Use realistic carriers as well as obvious attack strings. Seed a support ticket, PDF, email, web page or tool response with an instruction that would look plausible in production, then test whether the system treats it as data or authority. A filter that catches a literal 'ignore previous instructions' pattern can still miss a semantically equivalent attack, while an over-aggressive filter can break legitimate workflows. Measure both bypass and false-positive behaviour at the boundary that matters.

RAG: authorization belongs below the model

A retrieval layer should enforce the caller's authorization before documents are placed into model context. Prompt instructions such as 'only show this tenant's data' are not a substitute for access control. Testers should use controlled canary documents across users and tenants, manipulate metadata filters, test alternate retrieval routes and confirm that deleted or revoked data is no longer returned when policy says it should be gone.

Tools and agents: least privilege has to exist at runtime

Test denied actions and failure paths as deliberately as successful ones

Many control failures appear when the happy path breaks. Revoke a permission, return an API error, remove a tool, expire a token or make a downstream service unavailable. Then observe whether the agent stops safely, retries with a broader credential, falls back to a different tool, exposes internal error content or invents a successful outcome.

This matters because an approval prompt is only a control if the actual action cannot bypass it, and least privilege is only real if the system behaves safely when a requested action is outside the caller’s authority.

An agent can be well prompted and still be over-privileged. The backend must decide which identity is calling, which action is permitted and which parameters are acceptable. High-impact operations should have narrow permissions, explicit approval where appropriate, auditable logs and a safe recovery path.

Test both the obvious tool and the indirect route. If an agent cannot call an export function directly but can ask another tool to fetch a URL, write a file or send a message, the same business outcome may still be reachable through chaining.

Also test failure paths. Deny a tool, return a timeout, remove a preferred integration or give the agent a malformed response. Authorization should remain fixed by backend policy; the agent should not recover by selecting a broader tool, credential or destination than the user was originally entitled to use.

AI-specific testing still needs conventional security testing

AI features often sit behind ordinary REST or GraphQL APIs, object identifiers, storage buckets, OAuth tokens and cloud roles. Broken object-level authorization, SSRF, injection, insecure secrets and excessive IAM privileges can dominate the risk. A separate 'AI test' that ignores those controls gives a false sense of coverage.

How to handle non-determinism

Use test harnesses for coverage, but validate findings manually

Automated prompt suites are useful for breadth and regression. They are less useful when a detector treats an empty answer, a refusal or a semantically harmless variation as a security failure. High-impact findings should be manually reproduced and tied to an observable security consequence: unauthorized data, an unauthorized action, a trust-boundary bypass or unsafe downstream execution.

Record the model/provider, model version where available, system configuration, relevant tool set and test date. AI behaviour changes. Without that context, a later retest can appear inconsistent even when the application code did not change.

One successful or unsuccessful prompt is weak evidence in a probabilistic system. Record the model and configuration, repeat meaningful test cases, and report success rate or observed consistency where that changes the risk. Where model parameters are configurable, a low-variance baseline can help debugging, but production settings should also be tested because that is the behaviour users and attackers will encounter.

Score the consequence, not how clever the prompt looked

A dramatic jailbreak is not automatically a severe vulnerability. If the model can only produce text that a human must independently verify, the security impact may be limited. A plain-looking indirect instruction that causes the system to disclose another tenant’s document or invoke a privileged tool is far more serious.

Severity should therefore include the attacker-controlled input, the trust boundary crossed, the authority available to the application, the reliability of the exploit and the resulting impact. That makes AI findings comparable with the rest of the security programme instead of creating a separate severity language for prompts.

What leadership should measure

• Unauthorized data retrieval: can a user obtain information outside their entitlement?
• Privilege reach: which tools or APIs can the AI invoke, and under whose identity?
• Action safety: which operations require approval, and can that approval be bypassed or socially engineered?
• Persistence: can malicious context survive in memory, documents or vector stores?
• Containment: if one layer is manipulated, what prevents movement into the next?
• Detection: are retrieval, tool use, approvals and policy failures logged in a way incident responders can use?
• Repeatability: can the security team reproduce the important failures after prompts, models or controls change?

When to test

Test before a material AI feature reaches production, after adding new tools or data sources, after major model or orchestration changes, and after changes to identity or authorization boundaries. High-risk agents should also be re-tested periodically because their behaviour depends on connected systems that change independently of the model.

Common scoping questions

Q1: Is AI penetration testing the same as LLM red teaming?

Ans: No. Model red teaming is one part of the work. Application penetration testing also covers authorization, RAG, memory, tools, APIs, cloud controls and the path from model behaviour to business impact.

Q2: Does a strong system prompt stop prompt injection?

Ans: No. Prompts can reduce unwanted behaviour, but security boundaries should be enforced by deterministic controls such as authorization, tool permissions, schema validation, egress restrictions and approval gates.

Q3: Which framework should we use?

Ans: OWASP GenAI guidance, the OWASP Top 10 for Agentic Applications, NIST AI RMF/GenAI Profile and MITRE ATLAS are useful references. They should guide coverage, not replace architecture-specific threat modelling.

Scope an AI application penetration test

RedSecLabs tests the model, retrieval layer, APIs, identity boundaries and agent/tool paths as one system. The engagement is scoped around what the AI can actually access or change in your environment.

AI & LLM Penetration Testing Services  https://www.redseclabs.com/services/pentesting-services/ai-llm-penetration-testing-service