Testing agents is harder than testing functions — the path to a correct answer matters, not just the answer. An agent that gets the right result via an accidental sequence of tool calls is not a reliable agent.
Replace real tools and LLMs with deterministic mocks in tests. Fast, free, and reproducible — test agent logic without actually calling APIs.
import pytest
from unittest.mock import MagicMock, patch
from langgraph.graph import StateGraph
# ── Mock the LLM ──────────────────────────────────────────────
def make_mock_llm(responses: list[str]):
"""Returns a mock LLM that cycles through pre-defined responses."""
mock = MagicMock()
mock.invoke.side_effect = [
MagicMock(content=r, tool_calls=[]) for r in responses
]
return mock
# ── Mock tools ────────────────────────────────────────────────
def mock_search_tool(query: str) -> str:
"""Deterministic mock — always returns the same search result."""
return "Search result for: " + query
# ── Test agent behavior ───────────────────────────────────────
def test_agent_completes_without_tools():
"""Test that agent terminates when LLM doesn't call tools."""
mock_llm = make_mock_llm(["Here is the final answer: Paris"])
with patch("myapp.agent.model", mock_llm):
result = run_agent("What is the capital of France?")
assert "Paris" in result["messages"][-1].content
assert mock_llm.invoke.call_count == 1 # Called exactly once
def test_agent_uses_search_tool():
"""Test that agent correctly calls search and incorporates result."""
tool_response = MagicMock(content="", tool_calls=[{
"id": "call_001",
"name": "web_search",
"args": {"query": "population of Tokyo"},
}])
final_response = MagicMock(content="Tokyo has 13.96 million people.", tool_calls=[])
mock_llm = make_mock_llm([])
mock_llm.invoke.side_effect = [tool_response, final_response]
with patch("myapp.agent.model", mock_llm), \
patch("myapp.agent.web_search", mock_search_tool):
result = run_agent("What is the population of Tokyo?")
assert mock_llm.invoke.call_count == 2
assert "13.96 million" in result["messages"][-1].content| Level | Tests | Count | Run on |
|---|---|---|---|
| Unit (mocked) | State transitions, routing logic, tool input validation | ~30 | Every commit |
| Integration (real tools) | Tool calls work, API responses handled correctly | ~10 | Every PR |
| E2E eval (real LLM) | Agent achieves goal on golden dataset | ~20 tasks | Before release |
| Adversarial | Prompt injection, hallucination, loop-breaking inputs | ~10 | Before release |
A normal function returns the same output for the same input, so one assertion is enough. An agent has several sources of variation: the model samples its output, tools return live data, and a small change early in the run changes every later step. That has three consequences for how you test. First, push as much logic as possible into deterministic code (routing, parsing, validation, state updates) and test that with ordinary unit tests. Second, for the parts that involve a real model, assert on propertiessuch as "called the search tool" or "never called a write tool", not on exact wording. Third, run model-backed tests more than once and report a rate, not a single pass or fail.
Suppose each step of a five-step agent succeeds independently with probability 0.9. The chance the whole run succeeds is 0.9 × 0.9 × 0.9 × 0.9 × 0.9 = 0.9⁵ = 0.59. A task that looks "90% reliable" at each step fails about four times in ten end to end. Real steps are not fully independent, but the lesson holds: errors compound with the number of steps, which is why long agent chains need fewer, more reliable steps and checks between them.
Now the measurement side. If your golden set has 20 tasks and the agent passes 18, the pass rate is 90%, but the standard error is √(0.9 × 0.1 / 20) = √0.0045 = 0.067, so an approximate 95% interval is 0.90 ± 1.96 × 0.067, which is about 77% to 100%. A change that moves you from 18 to 19 passes is well inside that noise. To see improvements of a few points you need either many more tasks or repeated runs, and you should compare two versions on the same tasks so the task difficulty cancels out.
def pass_rate(task, check, runs=5):
"""Run one task several times; return the fraction of runs that pass."""
passes = 0
for _ in range(runs):
result, trace = run_agent_with_trace(task)
passes += 1 if check(result, trace) else 0
return passes / runs
# Gate a release on a rate, not on a single lucky run
assert pass_rate("Calculate 15% of $847", lambda r, t: "127.05" in r["messages"][-1].content) >= 0.8Most agent incidents are not wrong answers; they are loops, runaway cost, and bad tool arguments. Each can be tested deterministically with a mock model. A model that keeps requesting the same tool should be stopped by your step limit, and invalid arguments should be rejected before the tool runs:
import pytest
def test_agent_stops_at_step_limit():
looping = MagicMock(content="", tool_calls=[
{"id": "c1", "name": "web_search", "args": {"query": "again"}}
])
mock_llm = MagicMock()
mock_llm.invoke.return_value = looping # never produces a final answer
with patch("myapp.agent.model", mock_llm), patch("myapp.agent.web_search", mock_search_tool):
result = run_agent("Find something", max_steps=5) # your own step limit
assert mock_llm.invoke.call_count <= 5
assert result["stopped_reason"] == "max_steps" # the agent reports why it stopped
@pytest.mark.parametrize("bad_args", [{}, {"query": ""}, {"query": 123}])
def test_search_tool_rejects_bad_input(bad_args):
with pytest.raises(ValueError):
validate_search_args(bad_args)The names run_agent, max_steps, stopped_reason, and validate_search_args are stand-ins for your own code; the point is that the loop guard and the validator are ordinary deterministic functions and should have ordinary tests.
Live-model tests are slow, cost money, and drift. A useful middle layer records real model and tool responses once, stores them as fixtures, and replays them in later runs. This tests your orchestration against realistic data without the cost, and a failing replay means your code changed, not the model. Refresh the recordings on a schedule so they do not go stale, and keep a small set of live tests for the final check.
Put deterministic logic under conventional tests, mock the model to test control flow and failure handling, assert on tool-use properties through traces, and evaluate real runs on a golden set using rates and intervals. Errors compound over steps, small test sets are noisy, and the safest tests are the ones that prove the agent cannot do something it must never do.
You are now ready for the capstone: building a multi-agent research team that uses LangGraph, CrewAI, human-in-the-loop approval, and a full test suite.