LLMOps: RAG Pipelines, Evaluation, and Cost Control

An overview of the LLM Ops course, covering key topics, tools, and what you will build by the end.

Learn why treating prompts as code matters and how version control, templating, and A/B principles apply to prompt management.

Build a prompt versioning system using Git branches and Jinja2 templates, then run a basic A/B experiment comparing two prompt versions.

Learn how vector embeddings, ANN algorithms, and vector database architectures enable efficient semantic search over high-dimensional data.

Configure a local in-process Chroma vector database, ingest chunks embedded with OpenAI text-embedding-3-small, measure recall@5, then compare a local sentence-transformers embedder.

Learn the end-to-end RAG workflow (from query embedding to context-augmented generation) and the role of each component in reducing hallucinations.

Compose a functional RAG application with the raw OpenAI SDK and Chroma that retrieves relevant documents from a vector store and generates context-aware answers via an LLM API.

Learn how distributed tracing concepts apply to LLM applications and what key metrics (latency, token counts, and span hierarchies) reveal about application health.

Instrument a RAG application to send full execution traces to Arize Phoenix, then analyze latency and token usage across each pipeline step.

Learn the metrics and methodologies for evaluating LLM applications: reference-based metrics, judge models, golden sets, and regression detection.

Build an automated evaluation suite using RAGAS to measure retrieval relevance, answer faithfulness, and answer correctness against a golden test set, with a regression-detection gate.

Learn how LLM providers price API calls by token, how to count tokens programmatically, and the strategies used to aggregate and monitor costs at scale.

Instrument LLM API calls to capture token usage, log per-call cost data, and build a simple dashboard showing cost trends by day, model, and query type.

Learn how semantic caching uses vector similarity to serve cached LLM responses for semantically equivalent queries, reducing cost and latency.

Build a semantic caching layer that intercepts LLM calls, checks a vector store for similar cached queries, and stores new results: measurably reducing API calls.

Learn why specialized inference servers like vLLM outperform simple API wrappers and how techniques like PagedAttention and continuous batching maximize GPU throughput.

Learn how an LLM gateway centralizes model access, enforces organizational policies, secures API keys, and provides load balancing across multiple model providers.

Build a FastAPI-based LLM gateway that routes queries across model tiers, adds retry with exponential backoff, and abstracts a second provider behind a stubbed adapter.

Learn the safety risks specific to LLM applications (prompt injection, PII exposure, jailbreaks, and unsafe outputs) and the architectural patterns used to mitigate them.

Build input and output guardrails using LLM Guard to detect prompt injection, redact PII, and validate LLM output for safety and format compliance.

Learn how controlled experiments, statistical significance, and feature flagging are applied to evaluate competing prompt variants on real production traffic.

Implement a traffic-splitting system that routes live requests to two prompt variants, collects performance metrics, and applies a chi-squared test to determine the winner.

Learn how event-driven pipeline design and orchestration tools keep a RAG knowledge base current through automated, idempotent ingestion workflows.

Build an event-driven RAG pipeline with a filesystem watcher and an atomic blue/green index swap that chunks, embeds, and upserts new documents, with S3 + Lambda as the production analogy.

Learn how to profile each component of the RAG pipeline, identify latency bottlenecks, and apply optimization strategies including token streaming and hardware acceleration.

Profile a RAG pipeline to locate latency bottlenecks, measure token streaming against a blocking endpoint, and run a vector-search tuning sweep, measuring improvement at each step.

Recap of all LLM Ops skills covered, key takeaways, and guidance on next steps in the AI Ops Engineer program.