Last Updated on September 14, 2026
With insights from Dr. Josh Kalin, Director of AI at Integration Innovation, Inc. and AI Security Engineer Nanodegree instructor, and Kevin Carter, Security and Cloud Leader and AI Security Engineer Nanodegree instructor.
A car dealership deployed a ChatGPT-based sales assistant on their website. It was meant to answer customer questions and help move inventory. Within days, a user had used prompt injection to get the assistant to agree to sell an $80,000 vehicle for one dollar. The assistant confirmed the supposed deal in writing.
No database was breached. No firewall was bypassed. No conventional software exploit was required. The attack happened entirely through natural language.
“That’s where a lot of security engineers get caught,” says Josh Kalin, Director of AI at Integration Innovation, Inc. “They expect to look at the software, find a vulnerability, and exploit it. But with AI systems, an attacker doesn’t have to touch the software at all. They can manipulate how the model behaves, and that’s an entirely different attack surface.”
AI systems are now embedded in customer service tools, HR platforms, clinical workflows, financial systems, and agentic pipelines that take autonomous action on users’ behalf. The question of how to secure them is no longer theoretical. But the discipline required to do it isn’t traditional cybersecurity alone, and it isn’t machine learning engineering alone. It sits at the intersection of both, and most organizations don’t yet have people who can work there.
How do attackers exploit AI systems?
Traditional security red teaming has a familiar model: find the software, identify vulnerabilities, exploit them. The tools and frameworks built up over decades are highly effective at analyzing code, configurations, infrastructure, identity, and known adversary behaviors.
AI systems add another dimension. Some of their behavior emerges from training data, the prompts they receive, and the context they operate in rather than being explicitly defined by traditional program logic. That’s what makes some of the ways they fail so different.
Dr. Kalin outlines three categories where this shows up in practice.
The first is evasion. In white box attacks, an adversary who has access to a model’s architecture and gradients can craft inputs that cause predictable misclassification with high reliability. Researchers demonstrated this class of attack against computer vision technology used in Tesla vehicles: a small piece of tape altered a 35 mph speed sign in a way that caused the system to read it as 85 mph and begin accelerating accordingly. The attacker didn’t need access to the vehicle’s underlying software. They needed to understand how the model interpreted its environment.
Black box evasion is more widely applicable still. “A lot of people assume you need access to model internals to find valid attacks,” Dr. Kalin says. “Black box attacks are the opposite. If you have an API key and $20, you can find attacks that break that model.” By querying a system repeatedly and observing its outputs, an attacker can approximate aspects of the target’s behavior, develop attacks against a local surrogate, and test whether those attacks transfer to the real system.
The second category is data poisoning. Because large language models can train on massive datasets drawn from many sources, the integrity of that training data can be difficult to fully verify. Research involving the UK AI Security Institute, Anthropic, and the Alan Turing Institute found that as few as 250 malicious documents were sufficient to introduce a simple backdoor behavior across the models they tested, despite substantial differences in model size and training data volume. The finding demonstrates how a relatively small amount of poisoned data can create persistent model behavior, though the result does not necessarily generalize to every model or type of backdoor attack.
The third is prompt injection — manipulating the natural language instructions a model receives to make it ignore its intended behavior or perform unauthorized actions. Unlike SQL injection or command injection, which target deterministic software behavior, prompt injection targets the model’s interpretation of language and context. There is no static analysis pass that can catch every variation, because you cannot enumerate every way language can be turned against a system.
Why standard security frameworks aren’t enough for AI
Security teams deploying AI systems frequently reach for the tools they know: the OWASP LLM Top 10, built-in model guardrails, standard rate limiting. These aren’t wrong, but they’re not a security strategy on their own.
“The biggest misconception for junior security engineers is treating the OWASP LLM Top 10 as sufficient on its own,” says Kevin Carter, Security and Cloud Leader. “They run through the list and skip establishing traditional security principles first. Standard network controls, identity and access management, data access policies — all of that still has to be in place. You can’t think of the application as a standalone model.”
Carter identifies sensitive information disclosure as a common blind spot. Organizations focus on protecting what they’ve clearly labeled as sensitive, but AI systems — especially those built on retrieval-augmented generation — interact with a much broader surface of internal data. System addresses, internal documentation, access instructions: information that may appear harmless in isolation can become useful reconnaissance or expose resources a particular user was never intended to access.
The same blind spot extends into multi-agent systems, where the security surface expands with every agent, tool, and data source added to the pipeline. A compromised or manipulated orchestrator can send unsafe instructions to sub-agents that trust its output. A malicious or compromised sub-agent can pass unvalidated content back to an orchestrator. And because instructions between agents often travel as natural language, prompt injection vulnerabilities that affect user-facing systems can apply internally as well. Carter points to a real example: a publicly listed skill for an AI agent marketplace directed execution toward external infrastructure. When the associated instructions were executed, they ran an encoded payload, downloaded a binary, and stripped the system’s quarantine attributes — all without the user’s knowledge.
“Every external capability or source of context an agent uses has to be part of your threat model,” Carter says. “If an agent can download code, invoke a skill, execute a tool, or consume data from an external source, you need to apply the same supply chain principles you’d apply to software dependencies: provenance, validation, and least privilege.”
AI red teaming and defense: methodology, architecture, governance
Knowing what can go wrong is the starting point. Structured practice is what turns that knowledge into something an organization can actually use.
On the offensive side, that means a red team charter — the document that defines what you’re testing, what’s out of scope, the rules of engagement, and what success looks like. Dr. Kalin frames it as the difference between finding vulnerabilities and producing findings people can act on. “One of the most challenging things in adversarial AI is that people will find vulnerabilities and it’s like, okay great, now what? If you have the red team charter, you can actually describe the places you’re trying to harden and make sure you’re delivering something useful to the people who need to fix it.”
Threat modeling feeds into the charter. Knowing the model family or architecture narrows which attack vectors are actually valid — and which ones a red team can deprioritize. Public frameworks like MITRE ATLAS and the OWASP LLM Top 10 identify broad attack categories, but translating those into a specific list of valid threats for a specific deployed system takes analysis the frameworks can’t do for you.
On the defensive side, the foundation is determinism where possible. “The most reliable security controls are the ones you can enforce outside the model,” Carter says. “Tight IAM controls based on proper data classification. Role-based access so agents can only retrieve what they’re authorized to see. Least-privilege permissions around the tools they can invoke. The model can’t expose data it can’t reach, and it can’t take an action it doesn’t have permission to perform.” On top of that foundation, good AI security architecture separates content from authority. System prompts can define intended behavior, but authentication, authorization, data access, and tool permissions should be independently enforced. User input and externally retrieved content — including RAG documents — should be treated as untrusted rather than allowed to grant themselves authority through natural-language instructions.
Custom guardrails sit above all of that, tuned to the specific risk profile of each deployment. Built-in model safety systems are a starting point. They’re not calibrated to the sensitivity of your data, your users, or your threat landscape. Carter’s guidance: use threat modeling to identify the specific, highest-risk attacks for your organization, reduce what a successfully manipulated model can access or do, and build additional controls against the risks that remain — don’t assume the defaults cover them.
Governance extends this across time. AI systems change — models are updated, knowledge bases grow, agents acquire new tools. A governance framework defines how systems are assessed at deployment, how they’re monitored in production, and what review processes apply when they change. For high-stakes deployments in clinical, financial, or regulated environments, that framework also has to map to compliance requirements. It’s a document you produce before a regulator asks for it.
AI incident response: what’s different when the system is agentic
Contain, investigate, remediate — the structure of incident response doesn’t change for AI systems. What changes is what each step actually requires.
Adequate logging for an AI agent means being able to reconstruct the chain of events: user identity and role, relevant inputs and outputs where policy permits, which RAG documents were retrieved, every tool call the agent made, authorization decisions, agent-to-agent interactions, and the final model response or action. That telemetry also has to be handled according to its sensitivity — prompts and outputs can themselves contain credentials, personal information, regulated data, or proprietary content. Without sufficient visibility into those interactions, forensic analysis after an incident becomes extremely difficult. Many organizations currently log far less.
Containment in agentic systems carries a specific risk. “You have to contain the workflow, not just the agent you first identified as compromised,” Carter says. “If agents share credentials, tools, state, or trust relationships, isolating one component may leave the rest of the affected workflow operating.” And before deleting a compromised agent’s state, preserve the relevant evidence. The current state may be critical to understanding what happened.
Remediation requires more than patching the entry point. If an attacker introduced poisoned documents into a RAG knowledge base, those documents may have already shaped the system’s outputs and they may still be there. The knowledge base itself needs to be cleaned and reloaded. Credentials and session tokens may need to be rotated. Any actions the agent took during the compromise period need to be audited and, where possible, reversed.
The most important constraint on all of this is that someone has to understand the system well enough to know what normal looked like before the compromise. That’s not a logging problem. It’s a competency problem.
What this means for the security engineer’s role
The pattern across all of these scenarios is consistent. Securing AI systems requires understanding how attacks are designed against components that traditional security practices weren’t originally built to address — language models, training pipelines, vector databases, agent tool ecosystems — while also applying established security engineering principles to the applications and infrastructure around them.
Most security engineers have part of that picture. Most ML engineers have the other part. Neither role, as currently defined, necessarily covers both. That gap is where production AI can become vulnerable.
The AI Security Engineer Nanodegree program is built around closing it. Students build four projects across real cloud deployments — threat modeling and red teaming a Bedrock RAG agent, assessing a containerized image classifier, conducting a full security review of a RAG-enabled research agent, and completing a governance and compliance review of a clinical risk model — covering the full arc from first assessment to compliance sign-off.
If your organization is deploying AI systems and your current security team hasn’t red teamed a language model, assessed a multi-agent pipeline, or produced a governance review for a probabilistic system, that’s the gap this program is designed to address.
