AI Observability & Production Monitoring • Module A: Tracing & DebuggingLesson 1: LLM Tracing with LangSmith & Langfuse
You can't debug what you can't see. LLM tracing gives you a call-by-call view of what your AI system did: which model was called, with what inputs, what it returned, how many tokens it used, and how long each step took. Without it, production debugging is guesswork.
| Field | Why It Matters |
|---|---|
| trace_id | Links all spans in one user request — cross-reference with your app logs |
| model | Confirm the right model is being used, detect unexpected fallbacks |
| input_tokens / output_tokens | Cost tracking and budget alerting |
| latency_ms | Find the slow step in multi-step chains |
| inputs / outputs | Debug unexpected behavior — see exactly what went in and came out |
| status (ok / error) | Alert on LLM call failures, track error rates over time |
Best for: Best for LangChain/LangGraph applications. Native integration with LCEL chains and LangGraph agents.
# pip install langsmith
import os
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "ls_..."
os.environ["LANGCHAIN_PROJECT"] = "my-ai-app"
# That's it — all LangChain/LangGraph calls are now traced automatically
from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate
llm = ChatAnthropic(model="claude-haiku-4-5-20251001")
prompt = ChatPromptTemplate.from_template("Answer: {question}")
chain = prompt | llm
# This call is traced: inputs, outputs, latency, token usage, errors
result = chain.invoke({"question": "What is RAG?"})
# Traces appear in https://smith.langchain.com
# Drill into: each chain step, token counts, cost per run, latency breakdownTrace from day one.Retrofitting tracing into a production system is painful — every LLM call is buried in business logic. Add it at the start and you'll have baseline latency, cost data, and an error history before you ever need to debug a production incident.
Lesson 2 uses the data from your traces to build a cost tracking and optimization dashboard — turning token usage numbers into actionable cost reduction.