Blog · Chapter 08 · LLM deployment
Where the model runs, and why
Frontier APIs, cloud AI services, local models and dedicated GPUs, combined behind one model gateway.
The model is the most expensive and the most strategic dependency of an AI application. Chapter 8 compares the ways to run it and shows how to combine them.
The options
| Option | Strengths | Trade-offs |
|---|---|---|
| Frontier LLM API | Best quality on hard tasks, no operations, fast start | Data leaves your environment, per-token cost, provider dependency |
| Frontier agent app | Ready-made agents for knowledge work, extended with tools and skills | Less control over the workflow |
| Cloud AI service (Bedrock, Azure, Vertex) | Frontier models inside your cloud account and region | Cloud-specific setup, regional model availability |
| Local LLM (Ollama, LM Studio) | Free at the margin, fully private, good for development | Smaller models, limited throughput |
| Remote shared GPU | Open models without buying hardware | Variable availability and performance |
| Dedicated GPU (vLLM on your own servers) | Sovereignty, predictable cost at high volume | Operations, capacity planning, often lower quality on hard tasks |
Building on APIs vs extending agent apps
For process automation, such as claims intake or invoice processing, build on APIs: you need control over each step, evaluation and integration. For knowledge work, such as research, analysis and writing, extending an existing agent app with your tools, skills and data is often faster and better than building your own.
Check the terms, not just the benchmarks
Before sending business data to any provider, check the data-processing agreement, the retention and training policies, the processing regions and the certifications. Regulators will ask.
Combine options behind a gateway
A model gateway (for example LiteLLM, or a cloud AI gateway) exposes one endpoint and routes to several back-ends. It gives you:
- fallbacks when a provider is down or rate-limited;
- routing: small models for easy requests, frontier models for hard ones;
- one place for keys, budgets, logging and caching;
- freedom to switch providers without changing application code.
Key takeaways
- Frontier APIs give the best quality for the least effort.
- Self-hosting buys sovereignty at the price of operations.
- A model gateway adds fallbacks and avoids lock-in.