See every request, error and resource

AI applications have more moving parts than classic web apps: queues, workers, vector stores, GPUs and external model providers. Chapter 10 covers how to see what all of them are doing.

Instrument once with OpenTelemetry

OpenTelemetry gives you vendor-neutral metrics, logs and traces, all correlated by a trace ID. One user request can then be followed through the API, the queue, the worker, the retrieval step and the LLM provider. Auto-instrumentation covers web frameworks, HTTP clients and databases with a few lines of setup. A collector in the middle redacts sensitive fields and exports to your back-end.

What to monitor

• Events: business-level facts such as "claim processed", "approval requested" or "reply sent".
• Logs: structured JSON, correlated with traces, with personal data redacted.
• Infrastructure: CPU, memory, disk, container restarts, and GPU utilisation and memory for self-hosted models.
• Networking: latency and errors to every dependency, including each external LLM provider.
• Errors: rates, types, and which fallback path was taken.

Handle errors deliberately

LLM providers rate-limit, time out and occasionally go down. Build resilience in:

• retries with exponential back-off and jitter for transient errors;
• fallbacks to another model or provider;
• circuit breakers that stop hammering a failing dependency;
• quarantine and dead-letter queues for inputs that always fail.

Then monitor each mechanism. A fallback that fires constantly is an incident, not a success.

Dashboards and alerts

Build Grafana dashboards around the golden signals (latency, traffic, errors, saturation) plus your business KPIs. Alert on user-facing symptoms, such as a growing queue, rising error rates or slow responses, rather than every internal blip, and attach a runbook to every alert.

Local first, then choose a back-end

An all-in-one local stack (Grafana, Prometheus, Loki and Tempo in one container) is enough for development. For production, choose self-hosted or managed back-ends based on data control and budget.

Key takeaways

• Instrument with OpenTelemetry, correlated by trace ID.
• Handle errors deliberately, and monitor every fallback.
• Alert on user-facing symptoms, with runbooks.