Blog · Chapter 11 · Agent monitoring
See what the agent did, and why
Tracing LLM calls, tools and human decisions, catching silent failures, and closing the loop from feedback to fix.
Application monitoring tells you that a request took four seconds. Agent monitoring tells you why: which prompt version ran, what was retrieved, which tools were called, what the model answered and whether a human changed it. Chapter 11 covers this second layer.
Trace everything the agent does
A good agent trace records, for every run:
- each LLM call, with prompt version, model, tokens, latency and cost;
- each retrieval, with query, returned chunks and scores;
- each tool call, with arguments, result and errors;
- each guardrail decision and each human-in-the-loop action;
- metadata such as user, tenant, environment and release.
Tools such as LangSmith and Langfuse make these traces searchable, and OpenTelemetry's GenAI conventions make them portable.
Treat traces as sensitive data
Traces contain prompts, documents and answers, and often personal data. Mask fields, sample where volume is high, restrict access, and apply retention and deletion policies, including for erasure requests.
Catch silent failures
The most dangerous agent failures raise no exception: a confident wrong answer, a missed document, a tool called with the wrong customer ID. Catch them with:
- online evaluators that score a sample of production runs for groundedness, correctness and policy;
- user feedback: thumbs, edits, overrides and abandonment;
- behaviour metrics: steps per run, tool error rates, loops and escalations.
Agent KPIs
Track quality, cost and latency per run and per step, alongside adoption: acceptance rate, edit rate and time saved. These tell you whether the agent is getting better or worse after each release.
Close the loop
Monitoring only pays off when it changes the system. The loop is: negative feedback, then annotation by an expert, then a new case in the golden dataset, then an experiment with a fix, then a release gated by evaluations.
Key takeaways
- Trace every LLM call, tool call and human decision.
- Catch silent failures with online evaluators and feedback.
- Close the loop: feedback, dataset, experiment, release.