Blog · Chapter 14 · DevOps & LLMOps
Every change versioned, tested and reversible
Version control, CI with evaluation gates, prompt and model versioning, progressive releases and instant rollback.
In an AI application, behaviour changes when the code changes, and also when a prompt, a model version, an index or a guardrail changes. Chapter 14 brings DevOps discipline to all of them.
Version everything that changes behaviour
| Artefact | How |
|---|---|
| Code | Git, with branches and reviews |
| Prompts | Versioned files (for example YAML) with changelogs, not strings buried in code |
| Models | Pinned model IDs in configuration, never "latest" |
| Datasets | Golden datasets and training data with versions |
| Indexes | Index versions tied to the embedding model and chunking settings |
| Guardrails and infrastructure | Configuration and infrastructure as code |
Every production answer can then be traced to the exact versions that produced it.
Automated tests with a fake LLM
Unit tests should be fast, free and deterministic. Replace the LLM with a fake that returns recorded responses, and test parsing, validation, tools and workflow logic without network calls or cost.
Gate AI changes on evaluations
In CI, every change to a prompt, model or retrieval setting runs the evaluation suite. The pipeline compares quality, cost and latency with the current release and blocks the change if any metric falls below its threshold.
Release progressively
- Shadow: run the new version on real traffic without showing the result, and compare.
- Canary: send a small share of traffic to the new version, watch the metrics, then increase.
- Blue-green: switch all traffic at once, with the old version ready.
Make rollback instant
Each artefact needs its own rollback: redeploy the previous image, restore the previous prompt version, re-pin the previous model, or switch back to the previous index. Practise it before you need it.
Manage environments
Keep development, staging and production close to each other, with separate credentials and budgets and safe data: anonymised or synthetic data outside production.
Key takeaways
- Version everything that changes behaviour.
- Gate AI changes on quality, cost and latency.
- Release progressively and make rollback instant.