the production LLM engineering stack

There’s a wide gap between getting a model to do something clever in a notebook and shipping an LLM feature that survives real traffic, real money, and real adversaries. Most of what I’ve learned about crossing that gap never had a home — it lived in code reviews, incident postmortems, and half-finished notes. So I wrote it all down properly: The Production LLM Engineering Stack, a free technical guide, is now up on this site.

It’s organized as 34 topics across 9 layers, following a single request through the whole system: the harness and context you send, what actually happens inside the inference server (KV caches, prefill vs. decode, batching, quantization), making outputs and tool calls reliable, retrieval you can measure, evals and observability, security and multi-tenant isolation, and the named failure modes that page you at 2am. Every topic ends with pitfalls and a concrete “what to actually do”, and the whole thing is source-linked — about 32k words, with a filterable glossary and decision checklists at the end.

It assumes you can already call an LLM API and wants to get you to the point where you can run one on-call. Read it front to back or jump straight to whatever’s currently on fire — the sidebar is built for both. If you find something wrong or outdated, tell me; it’s a living document and I intend to keep it verified.