Advanced~8 hours6 lessons

AI Observability & Production Monitoring

Build full-stack visibility into AI systems: distributed tracing, cost dashboards, quality drift detection, SLOs, and incident response playbooks for LLM-powered applications.

What you'll build: an end-to-end tracing and budget dashboard

Start Course

Curriculum

Module A: Tracing & Debugging

3 lessons

Lesson 1: LLM Tracing with LangSmith & Langfuse

Instrument LangChain, LangGraph, and raw SDK calls to produce distributed traces. Use LangSmith and Langfuse to visualize full agent execution trees, token counts, and latency breakdowns.

70 min

Lesson 2: Token Cost Tracking & Budget Enforcement

Track token spend per user, per feature, and per model. Build a cost dashboard, set per-request budgets, and implement automatic cost alerts before monthly bills spike.

65 min

Lesson 3: Structured Logging for LLM Applications

Design a logging schema that captures the full context of every LLM call: model, tokens, latency, prompt version, and outcome. Ship logs to a queryable store (BigQuery, ClickHouse) for analytics.

55 min

Recommended Watch

LLM Observability Explained

Trace, debug, and understand LLM behavior with structured logging and distributed tracing.

Arize AI14:26

Module B: Monitoring & Response

3 lessons

Lesson 4: Quality Drift Detection & Alerting

Detect when your AI system silently degrades — from model updates, data distribution shifts, or prompt regressions. Set up rolling quality monitors that alert before users complain.

65 min

Lesson 5: SLOs & SLAs for AI Systems

Define meaningful Service Level Objectives for AI systems: not just uptime, but quality, latency, and hallucination rate. Build error budgets that tell you when to stop shipping and fix reliability.

60 min

Lesson 6: Capstone — Build an AI Observability Stack

Instrument a multi-agent AI application end-to-end: add distributed tracing, cost tracking, quality sampling, and drift alerts. Deploy a Grafana dashboard and write a runbook for on-call engineers.

120 min

Recommended Watch

Monitoring ML Models in Production

Detect data drift, quality degradation, and latency spikes in production AI systems.

Weights & Biases18:52

Course Quiz

5 questions · ~5 min · Pass 4/5 to unlock your badge

Start Quiz →