Home / University / Security for AI Code / Article

Where Attackers Get Into Your AI App

Vibe Coder · 1.11 · Vibe Coding Fundamentals

Back to Vibe Coder

Why This Matters

The security field's ranked list of the ten most common web-app flaws — known as the OWASP Top 10 — covers the application. It does not cover what happens when the model itself is the attack vector. Every AI-augmented app introduces four new categories that don't exist in classic web threat modeling: malicious prompts, untrusted content the model treats as instructions, tampered agent state, and over-scoped tool access. None of these get caught by Snyk, Semgrep, or a code-review pass — because the vulnerability isn't in your code, it's in what the model decides to do with your code.

The Problem

Your app passes the OWASP sweep. No typed input executing as a database command — the attack known as SQL injection. No attacker script smuggled into pages your users see — known as XSS. No hardcoded secrets. You ship. A week later a customer pastes a support ticket into your AI summarizer, the summarizer reads instructions hidden in that ticket, and your agent emails the customer's purchase history to an attacker-controlled address.

Nothing in your code was wrong. The model did exactly what it was told — just not by you. Smuggling instructions to a model inside ordinary-looking input is an attack called prompt injection, and the standard security exercise — writing down what can attack your app and how, a document called a threat model — wasn't built for it. Classic threat models assume input is data. AI threat models have to assume input is also instructions.

This is the gap module 09's other guides don't close. The OWASP checklist asks "can this input cause SQL to misbehave?" The AI attack surface asks "can this input cause the model to misbehave?" Different question, different controls, different audit.

The Core Insight

Every path an attacker can use to reach your app is called an attack surface — and an AI-augmented app has four of them, not one. Threat-model each one separately.

  1. Direct prompt injection — the user types the malicious instructions themselves.
  2. Indirect prompt injection — the malicious instructions arrive inside content the model is asked to process (a document, a webpage, an email, a tool result).
  3. Agent instruction tampering — the model's own scratchpad, memory, or tool outputs get poisoned, so a future turn acts on adversary-controlled state.
  4. Tool / data-access scope — the model has more capability than the request needs (read a different user's data, call a destructive API, exfiltrate to a webhook).

The four are listed in order of how often they get missed. Direct injection is on most teams' radar. Indirect injection is the one that ships to prod. Agent tampering and tool scope are the ones that turn a small breach into a big one.

Surface 1: Direct Prompt Injection

A user sends input designed to override the system prompt or change the model's behavior. The classic "Ignore previous instructions and...", but also the subtle versions: roleplay framing, fake delimiters that look like system markers, encoded payloads that decode to instructions.

What to look for

Controls

The Two-Channel Rule

Anything authored by you = instructions. Anything authored by anyone else = data. If those two channels share a single string, you have a prompt-injection vulnerability the same way SQL has injection when you concatenate user input into a query.

Surface 2: Indirect Prompt Injection (Untrusted Content)

This is the one that catches teams who handled #1. The user is not malicious — they paste a document, link a URL, attach a CSV. The document is malicious. The model reads it, treats the embedded instructions as orders, and acts.

Examples that have shipped to production at real companies:

Controls

Surface 3: Agent Instruction Tampering

Long-running agents accumulate state: scratchpads, memory, tool results, prior turns. Every entry in that state is a potential injection point for the next turn. The model that wrote the entry trusted what it wrote. The model that reads it on turn 12 has no way to tell which words came from you and which came from an upstream poisoned source.

What to look for

Controls

Surface 4: Tool & Data-Access Scope

The damage from #1, #2, and #3 is bounded by what the model can do. A jailbroken chatbot that can only read public docs is embarrassing. A jailbroken agent with shell access, an SES key, and read-access to every customer's row is a breach.

Most over-scoping comes from convenience: it was easier to give the agent the service-role Supabase key than to wire per-user RLS-aware tokens. Easier to expose execute_sql(query) than 12 narrow read functions. Easier to let one MCP server cover ten capabilities than to split it.

Controls

The Threat-Modeling Pass

Before any AI-augmented feature ships, walk it once through the four surfaces. Twenty minutes per feature, and the questions are the same every time:

SurfaceQuestionControl
1. Direct injection Can the user override the system prompt by what they type? Structural channels; envelope user input; ignore authority claims.
2. Indirect injection What content does the model read that the user didn't write themselves? Treat as untrusted; sanitize; confirm before any tool action.
3. Agent tampering Whose words end up in memory, scratchpad, or tool results? Provenance tags; sanitize on write; bounded replay.
4. Tool / data scope What's the worst the model can do in a fully-jailbroken state? Narrow tools; per-call auth; egress allowlist; humans gate writes.

The Pre-Production Threat Model Prompt

Use this in the same session that just generated the feature, before merging:

"Threat-model this AI-augmented feature against four surfaces:

1. DIRECT PROMPT INJECTION
   - Where does user input flow into a model call?
   - Is it structurally separated from instructions, or concatenated?
   - What's the impact if a user pastes an injection payload?

2. INDIRECT PROMPT INJECTION
   - What external content does this feature feed to the model?
     (documents, URLs, tool outputs, search results, memory)
   - For each: what happens if that content contains hidden
     instructions telling the model to do something else?

3. AGENT TAMPERING
   - Does the feature store anything the model later re-reads?
   - Who can write to that store?
   - Does the model know which entries are trusted?

4. TOOL / DATA-ACCESS SCOPE
   - List every tool the model can call from this feature.
   - For each: what's the worst-case action in a fully-compromised
     state?
   - Which tools have destructive or exfiltrating capability?
   - Which require human confirmation before executing?

For each surface, rate exposure (None / Low / Med / High) and
propose the smallest control that closes it."

Failure Patterns

"Just tell the model not to follow injected instructions."

A system-prompt instruction is not a control. Models comply or resist statistically, not absolutely. Anything safety-critical must be enforced in the surrounding code — structural channels, sanitization, egress allowlists, human gates.

"It's an internal tool, the users are trusted."

The users are trusted. The documents they paste are not. Surface 2 is the indirect-injection surface; internal users still feed external content into the model.

"We use a small model, it can't really be jailbroken."

Smaller models are easier to jailbreak. Capability is the question for #1; scope is the question for #4. A small model with broad tool access is the worst combination.

"Our MCP server is read-only, so scope doesn't matter."

Read-only against what? An MCP server with read access to every user's data still enables cross-account exfiltration. Scope is about how far the damage from the worst read or write can spread — known as the blast radius — not about the verb.

Key Takeaways

What To Remember

Related Guides

This is the AI-specific layer of the Module 09 security trio. The other parts: