AI Ops Engineer

Get oriented with the ML Ops course, its goals, prerequisites, and the tools you will use throughout. Introduce the big picture and the course roadmap

Collect, store, and version datasets for reproducible ML pipelines with traceability from raw data to training-ready sets, and validate them to detect ML-specific quality concerns.

Ingest a dataset from a public API, store it locally, track dataset versions using DVC and Git Validate the dataset using Great Expectations

Learn how feature stores solve train-serve skew by providing consistent feature definitions and serving.

Define feature views in Feast, generate point-in-time training datasets, and retrieve features for inference.

Learn the principles of experiment tracking and how MLflow organizes parameters, metrics, and artifacts.

Instrument a training script to log parameters, metrics, and models to MLflow, then compare runs in the UI.

Learn CI/CD principles applied to ML, including triggers, orchestration, and automated retraining strategies.

Create a CI/CD pipeline that detects new data, retrains a model and validates it

Learn how model registries provide versioning, stage transitions, and governance for production ML models.

Register models, transition them through lifecycle stages, and build applications that load production models.

Learn what containers are, why they solve environment inconsistency, and how Docker packages ML model APIs.

Package a trained model and its FastAPI service into a Docker container, then build, run, and push the image.

Learn how to design model test suites that validate performance, robustness, and fairness before deployment.

Create automated test suites that evaluate model performance on subpopulations and block flawed deployments.

Learn how to deploy containerized models as APIs in production

Deploy a containerised model API to AWS, verify endpoint responses, and apply a cost optimization strategy to reduce serving costs.

Learn how to automate deployment pipelines for ML models, including CI/CD principles, canary release patterns, and automated rollback strategies

Build CI/CD pipeline that automatically deploys a containerized ML model from registry

Learn to monitor deployed ML model APIs for latency, error rates, resource usage and trigger alerts

Implement monitoring for a deployed ML model API, including metrics collection, dashboards, and alerting.

Learn the statistical foundations of data drift, how it differs from concept drift, and why monitoring matters.

Generate drift reports comparing training and production data, identify drifted features, and set up alerts.

Learn explainability methods like SHAP and LIME, the difference between local and global explanations, and their limits.

Create an API endpoint that returns SHAP-based feature importance explanations for individual predictions.

Learn fairness metrics like demographic parity and equalized odds, and how to evaluate bias across groups.

Assess model fairness across subgroups, generate fairness reports, and set up automated regression alerts.

Learn how managed cloud ML services integrate into a cohesive ML Ops lifecycle from training to monitoring.

Create a full ML Ops pipeline on SageMaker with automated training, deployment, auto-scaling, and monitoring.

Review the skills gained throughout the ML Ops course and explore next steps for applying them in production.

An overview of the LLM Ops course, covering key topics, tools, and what you will build by the end.

Learn why treating prompts as code matters and how version control, templating, and A/B principles apply to prompt management.

Build a prompt versioning system using Git branches and Jinja2 templates, then run a basic A/B experiment comparing two prompt versions.

Learn how vector embeddings, ANN algorithms, and vector database architectures enable efficient semantic search over high-dimensional data.

Configure a local in-process Chroma vector database, ingest chunks embedded with OpenAI text-embedding-3-small, measure recall@5, then compare a local sentence-transformers embedder.

Learn the end-to-end RAG workflow (from query embedding to context-augmented generation) and the role of each component in reducing hallucinations.

Compose a functional RAG application with the raw OpenAI SDK and Chroma that retrieves relevant documents from a vector store and generates context-aware answers via an LLM API.

Learn how distributed tracing concepts apply to LLM applications and what key metrics (latency, token counts, and span hierarchies) reveal about application health.

Instrument a RAG application to send full execution traces to Arize Phoenix, then analyze latency and token usage across each pipeline step.

Learn the metrics and methodologies for evaluating LLM applications: reference-based metrics, judge models, golden sets, and regression detection.

Build an automated evaluation suite using RAGAS to measure retrieval relevance, answer faithfulness, and answer correctness against a golden test set, with a regression-detection gate.

Learn how LLM providers price API calls by token, how to count tokens programmatically, and the strategies used to aggregate and monitor costs at scale.

Instrument LLM API calls to capture token usage, log per-call cost data, and build a simple dashboard showing cost trends by day, model, and query type.

Learn how semantic caching uses vector similarity to serve cached LLM responses for semantically equivalent queries, reducing cost and latency.

Build a semantic caching layer that intercepts LLM calls, checks a vector store for similar cached queries, and stores new results: measurably reducing API calls.

Learn why specialized inference servers like vLLM outperform simple API wrappers and how techniques like PagedAttention and continuous batching maximize GPU throughput.

Learn how an LLM gateway centralizes model access, enforces organizational policies, secures API keys, and provides load balancing across multiple model providers.

Build a FastAPI-based LLM gateway that routes queries across model tiers, adds retry with exponential backoff, and abstracts a second provider behind a stubbed adapter.

Learn the safety risks specific to LLM applications (prompt injection, PII exposure, jailbreaks, and unsafe outputs) and the architectural patterns used to mitigate them.

Build input and output guardrails using LLM Guard to detect prompt injection, redact PII, and validate LLM output for safety and format compliance.

Learn how controlled experiments, statistical significance, and feature flagging are applied to evaluate competing prompt variants on real production traffic.

Implement a traffic-splitting system that routes live requests to two prompt variants, collects performance metrics, and applies a chi-squared test to determine the winner.

Learn how event-driven pipeline design and orchestration tools keep a RAG knowledge base current through automated, idempotent ingestion workflows.

Build an event-driven RAG pipeline with a filesystem watcher and an atomic blue/green index swap that chunks, embeds, and upserts new documents, with S3 + Lambda as the production analogy.

Learn how to profile each component of the RAG pipeline, identify latency bottlenecks, and apply optimization strategies including token streaming and hardware acceleration.

Profile a RAG pipeline to locate latency bottlenecks, measure token streaming against a blocking endpoint, and run a vector-search tuning sweep, measuring improvement at each step.

Recap of all LLM Ops skills covered, key takeaways, and guidance on next steps in the AI Ops Engineer program.

An orientation to the AgentOps course: the RealThor agent you will operate, the four chapters of the course, and why agent operations differs from general LLMOps.

Learn why agent behavior is defined by more than code and how to version prompts, tools, and configurations for reproducibility.

Apply structured project layout and environment management to an agent so any past behavior can be reproduced and any change can be traced.

Learn how to design reproducible evaluation suites that establish trust in agent behavior and detect regressions across versions.

Define a benchmark task set with success criteria, build an automated test harness, and produce a regression report comparing agent versions.

Learn how agents run under real conditions (on-demand, triggered, or scheduled) and how CI/CD pipelines enforce quality before code reaches production.

Package an agent as a containerized CLI with Docker Compose and a click interface, using an ops.sh script that runs the evaluation suite as a quality gate before the image is built.

Learn the control layer between an agent and the external world that constrains what the agent can do, regardless of what the model reasons.

Build guardrails as LangChain AgentMiddleware that intercept every agent step on the RealThor agent: enforcing role-based access control, regional data access rules, and output redaction.

Learn why agents that execute code require isolated environments and how container-based sandboxing separates the reasoning layer from execution.

Build an in-process code execution tool that validates agent-generated Python with the ast module, runs it against restricted globals over pandas DataFrames, and enforces a thread-based timeout.

Learn the design principles for systems that pause at high-risk decisions, treating human oversight as a designed control mechanism, not a fallback.

Build a LangGraph agent that pauses before a high-risk action, notifies a reviewer, and routes execution correctly based on approve or reject signals.

Learn the signals that reveal when something is wrong in a deployed agent and how dashboards turn run data into operational visibility.

Add structured run-level logging to a deployed agent and build a pure-Python monitor that generates a self-contained report detecting failures, cost overruns, and latency degradation.

Learn how to reconstruct the exact conditions of a failed agent run using state checkpoints so failures that cannot be reproduced can be diagnosed.

Configure a durable SqliteSaver checkpointer, then reconstruct the context of a failed agent run from its newest persisted snapshot and its runs.jsonl log to diagnose what happened.

Learn how tracing turns debugging from guesswork into understanding by revealing the full sequence of thoughts, tool calls, and intermediate results.

Instrument a LangGraph agent with Langfuse via a LangChain callback handler and use the resulting span tree to inspect the agent's reasoning and tool calls.

Learn how context scoping and session isolation prevent state leakage and loss when a single agent system serves many users simultaneously.

Build a multi-tenant LangGraph agent that isolates each user's state with a per-(user, session) thread key backed by a SQLite checkpointer, verified with thread-key and ownership tests.

Learn how MCP standardizes connections between AI agents and external tools, enabling structured, scalable communication across systems and other agents.

Connect an AI agent to a provided FastMCP server that exposes custom tools by loading them through langchain-mcp-adapters, running as an unauthenticated development service.

A capstone reflection on the 12 AgentOps skills and how versioning, deployment, control, observability, and scale combine in production agentic systems.