AgentOps: Deploying, Guarding, and Monitoring AI Agents

An orientation to the AgentOps course: the RealThor agent you will operate, the four chapters of the course, and why agent operations differs from general LLMOps.

Learn why agent behavior is defined by more than code and how to version prompts, tools, and configurations for reproducibility.

Apply structured project layout and environment management to an agent so any past behavior can be reproduced and any change can be traced.

Learn how to design reproducible evaluation suites that establish trust in agent behavior and detect regressions across versions.

Define a benchmark task set with success criteria, build an automated test harness, and produce a regression report comparing agent versions.

Learn how agents run under real conditions (on-demand, triggered, or scheduled) and how CI/CD pipelines enforce quality before code reaches production.

Package an agent as a containerized CLI with Docker Compose and a click interface, using an ops.sh script that runs the evaluation suite as a quality gate before the image is built.

Learn the control layer between an agent and the external world that constrains what the agent can do, regardless of what the model reasons.

Build guardrails as LangChain AgentMiddleware that intercept every agent step on the RealThor agent: enforcing role-based access control, regional data access rules, and output redaction.

Learn why agents that execute code require isolated environments and how container-based sandboxing separates the reasoning layer from execution.

Build an in-process code execution tool that validates agent-generated Python with the ast module, runs it against restricted globals over pandas DataFrames, and enforces a thread-based timeout.

Learn the design principles for systems that pause at high-risk decisions, treating human oversight as a designed control mechanism, not a fallback.

Build a LangGraph agent that pauses before a high-risk action, notifies a reviewer, and routes execution correctly based on approve or reject signals.

Learn the signals that reveal when something is wrong in a deployed agent and how dashboards turn run data into operational visibility.

Add structured run-level logging to a deployed agent and build a pure-Python monitor that generates a self-contained report detecting failures, cost overruns, and latency degradation.

Learn how to reconstruct the exact conditions of a failed agent run using state checkpoints so failures that cannot be reproduced can be diagnosed.

Configure a durable SqliteSaver checkpointer, then reconstruct the context of a failed agent run from its newest persisted snapshot and its runs.jsonl log to diagnose what happened.

Learn how tracing turns debugging from guesswork into understanding by revealing the full sequence of thoughts, tool calls, and intermediate results.

Instrument a LangGraph agent with Langfuse via a LangChain callback handler and use the resulting span tree to inspect the agent's reasoning and tool calls.

Learn how context scoping and session isolation prevent state leakage and loss when a single agent system serves many users simultaneously.

Build a multi-tenant LangGraph agent that isolates each user's state with a per-(user, session) thread key backed by a SQLite checkpointer, verified with thread-key and ownership tests.

Learn how MCP standardizes connections between AI agents and external tools, enabling structured, scalable communication across systems and other agents.

Connect an AI agent to a provided FastMCP server that exposes custom tools by loading them through langchain-mcp-adapters, running as an unauthenticated development service.

A capstone reflection on the 12 AgentOps skills and how versioning, deployment, control, observability, and scale combine in production agentic systems.