Last Updated on September 25, 2026
Most engineers approaching their first production LLM system bring a mental model that works well for software: test your inputs and outputs with assertions, check your code into git, roll back if something breaks, and rely on system prompts or configuration files to enforce behavior. These habits are not wrong — they just do not account for what makes AI systems different. AI systems are non-deterministic. Same input, different output. Model behavior changes between versions, sometimes without notice, and can degrade over long conversations. Guardrails that “work in testing” can be bypassed through attacks that traditional security frameworks were never designed to catch. And agents that run reliably as a single user process on your laptop will behave differently — sometimes dangerously differently — when deployed at scale across concurrent users, webhooks, and batch jobs.
Jeff Chen, a practitioner who builds and tests AI deployments at SecureIQLab and teaches LLM deployment and security in Udacity’s new AI Operations Nanodegree program, and Henrique Santana, Principal ML Engineer for MLOps at Dell Technologies and also an instructor on the program, spend their professional lives working in that gap between what software ops looks like and what AI ops actually requires. Their observations point to the same underlying problem: the instincts engineers develop for deterministic systems will fail them if they apply those instincts unchanged to LLMs and agents. Here is where those instincts fail, and what to do instead.
Why production LLM deployment is not a scaled-up prototype
The first gap shows up early. You load a model into a Python process, throw a Flask endpoint in front of it, and it works. It gives you tokens back. It behaves correctly on your laptop. The temptation is to assume a server will do the same thing with more traffic.
It will not.
“When you are using these models for production, there are a lot more variables than running a model locally,” Jeff Chen explains. “You’re calling the model over a public cloud network, with variability you don’t control. You need to handle many different types of exceptions, manage retries and graceful fallback, and make sure you don’t get into a retry loop.”
The solution is not just more compute — it is a different architecture. A centralized API gateway gives the organization a common point for security, logging, and caching. It lets teams enforce internal rate limit policies and implement graceful degradation: when the primary model hits limits, the system falls back to a lower-capability model and keeps running, instead of failing completely. Without a gateway, every team in the organization manages its own retry logic, its own logging, and its own direct connections to model providers. Usage is invisible. Configuration errors compound. Problems that could be caught in one place multiply across dozens of integrations.
The same logic applies to observability. Jeff notes that when time-to-first-token is slower than expected, engineers need to know which layer is the actual bottleneck before they can optimize anything. “If you spend your time optimizing a stage that contributes little to the latency, you will not have fixed your problem.” Instrumentation is not optional in production AI systems; it is the mechanism that makes diagnosis possible.
Why system prompts are not guardrails
One of the most common misconceptions in production AI deployment is that a well-written system prompt is sufficient for safety. Tell the model not to reveal sensitive information. Tell it not to follow injected instructions. If it listens in testing, you might assume you are done.
Jeff Chen is direct about this assumption: “Relying on just adding prompts to the LM that you’re safeguarding against, hoping that it will follow the rules, is very dangerous.”
The reason is that the model being protected is also the model being attacked. Prompt injection and jailbreaking attacks exploit this directly. A well-known example is EchoLeak, a 2025 vulnerability in Microsoft 365 Copilot. An attacker sent an ordinary-looking email. When Copilot later pulled that email in as context, the text inside it acted as instructions, and Copilot leaked the user’s data through a link in its own response. Nobody clicked anything. A system that defends against injection by relying on the primary model’s judgment has no defense layer that sits outside what the attacker is targeting.
A stronger pattern is to put independent guardrails outside the primary model: a separate classifier on the input, and separate checks on the output. No single layer is a wall. EchoLeak got past Microsoft’s own injection classifier. Defense in depth only counts when the layers fail in different ways. The same principle applies to RAG. If an attacker gets a document into your knowledge base, the malicious text reaches the model as retrieved context and never touches the input filter. So Jeff recommends screening documents when they enter the index as well as user input at the gateway, checking outputs before they leave, and treating data and instructions as separate pipelines: “Separate data from instructions so the system gets a clear indication of how to process each one before it hands off.”
There is also a more subtle operational problem with guardrails: tuning them for one audience often means tuning them incorrectly for another. Jeff worked on an interview system that needed different content moderation thresholds depending on whether the respondent was a student or a teacher. “If a teacher said there were students who were abusive, you want to let that through — you want to capture that there’s a problem. But the same language from a student should be flagged for review.” Context-aware filtering requires understanding the actual use case, not just the general behavior of the model.
Versioning AI systems: more than a git commit
Software engineers are accustomed to a clean rollback: check out last week’s commit and the system reverts to how it was. In AI systems, that assumption breaks down quickly, because the components that shape model behavior are not all code.
In AI systems, the behavioral state of the application extends beyond source code. Prompts, model versions, retrieval indices, agent state, memory stores, tool definitions, and runtime configuration can all change system behavior. These artifacts may evolve on different lifecycles and therefore need to be versioned and governed independently. And the stakeholders who need to change these components — product managers, business owners, even executives — may not have any relationship with the CI/CD pipeline.
Henrique describes a real example: a company building a financial services product where the CEO needed to test different prompt versions. “He’s not a software engineer, so it’s not up to him to every time go through CI/CD to push all the code. He just wants to change the prompt version.” The system had to allow component-level updates — prompt versions, configuration changes, memory updates — without triggering a full build and deployment cycle. This requires intentional decoupling. Prompts live in a versioned store separate from the application codebase. Configurations and memory databases are tracked as their own artifacts. The ability to say “revert just the prompt to last week’s version” without touching the surrounding system is a design constraint, not an afterthought.
Evaluation for non-deterministic systems
Traditional software testing often asks whether a known input produces an expected result. LLM and agent evaluation frequently requires a different treatment: multiple outputs may be acceptable, behavior can vary across repeated executions, and quality may need to be measured across several dimensions rather than through a single exact-match assertion.
“When you assert something, you expect only two values — true or false,” Henrique says. “But when you’re evaluating, you have a range of values, and there is a baseline. And every time you evaluate, you have a purpose.”
Non-determinism also means that running a test suite once is not enough. A 10% failure rate will often go undetected in a single run. The same evaluation needs to run multiple times to surface reliability problems. A system that appears to pass four consecutive runs might be failing one in ten attempts in production.
Henrique recommends starting evaluation with regression testing: the new version of the system must perform at least as well as the previous one. “You start with a baseline. Before asking whether the new system is good in isolation, you ask what changed relative to the version you already trust. Any regression on a critical metric should be visible, measured, and justified.” From there, evaluation expands to cover new behaviors and components introduced in the updated version.
His framework for triage has three layers: input and output (black-box evaluation — does the system produce the right output given this input, and can you explain it to a stakeholder?); individual component assessment (once the black-box layer is passing, look inside at each component); and trajectory evaluation (analyze the path the agent took to reach an output — was it efficient, did it consume more tokens than expected, did it take an unexpected decision path?).
The third layer — trajectory evaluation — is where tools like LangSmith, Langfuse, or Arize Phoenix become necessary. Traces show the sequence of decisions an agent made during a session. They also surface missing features: if a user keeps asking the agent for something it cannot do, the traces record the dead ends. Those dead ends are often the clearest signal that new tools or data flows need to be added.
Monitoring versus observability
In traditional infrastructure, monitoring tells you whether a system is up or down. For AI agents, that is the minimum viable signal, not a comprehensive operational picture.
“Monitoring is about up or down,” Henrique Santana explains. “Observability — which includes logs and traces — is what you need to understand why something failed.” An agent that appears to be running but is stuck in a retry loop, exhausting its token budget, or generating semantically incorrect outputs will pass a basic health check. It takes observability to catch it.
Replayability is the capability that makes this tractable. When an agent session fails, a replay of the state and the trace of decisions lets engineers examine the exact sequence of events that led to the failure. For failures caused by non-deterministic model behavior or external vendor updates, the replay may not reproduce the exact failure — but it provides enough visibility to identify where the logic broke down and where closer monitoring is needed.
Data isolation is a related concern that Henrique flags specifically for multi-user deployments. In agentic systems that serve multiple tenants, failing to implement proper row-level access controls means one user’s semantic search can surface another user’s data. “These issues come from a lack of traditional software engineering practices being applied,” he notes — surrogate keys and tenant isolation are not optional in multi-user deployments.
What this means for the AI Ops engineer
The AI ops engineer is not a DevOps engineer with a new set of tools. The role requires a fundamentally different mental model for reliability, testing, and safety.
Jeff’s three core practices for production LLM systems: build evaluation and system tests before features (teams that do this ship faster), instrument every layer of the solution to support observability, and plan for the model to change underneath you. “LLM systems are non-deterministic and subject to change,” he notes. Building a system that handles that non-determinism gracefully is the engineering challenge — not just deploying a model and hoping it behaves.
Henrique’s core takeaway for long-term reliability is that AI operations require control over the system’s full behavioral state. Prompts, model versions, configurations, memory, retrieval state, and tool definitions need explicit versioning; evaluation needs baselines and repeated measurement; and observability needs to preserve enough execution context to reconstruct how an agent reached a result.
The new AI Ops Engineer Nanodegree program from Udacity teaches these skills through three courses: MLOps Foundations (automated, reproducible pipelines for traditional ML), LLM Ops (RAG pipelines, tracing, evaluation, cost monitoring, caching, gateways, guardrails, and latency), and AI Agent Operations (versioning strategies, evaluation frameworks, monitoring, and human-in-the-loop design for agentic systems).
