You do not need to derive the attention equation to architect LLM systems — but you do need to know which knobs it turns. Almost every design decision you will make in an enterprise AI system — context size, caching strategy, model choice, cost model, latency budget — is a downstream consequence of a small set of mechanical facts about how these models are built and served. This article gives you exactly those facts, at the depth an architect needs: enough to argue with the MLOps team, explain a bill to finance, and spot a bad design on a slide in thirty seconds.
We will cover how text becomes tokens and embeddings, what self-attention actually buys you, the three training stages behind every model you can buy, why long prompts are prefill-bound, how the KV cache turns memory into latency, what sampling parameters really control, and how the economics of inference should shape your architecture. Where it helps, I will give you the "explain it to a VP" version too.
Executive Summary
- Everything is priced and sized in tokens — cost, latency, context limits. Know your token economics before you design anything.
- Self-attention is O(n²) in context length — that is why long prompts are expensive to prefill even when the answer is short.
- Latency splits into TTFT (time to first token, set by prefill) and TPOT (time per output token, set by decode). They have different causes and different fixes.
- The KV cache trades memory for speed — it is the reason long-context serving eats VRAM and why prefix caching exists.
- Models you buy are the product of three stages — pre-training, SFT, alignment — and you can only influence the last two.
- Sampling parameters (temperature, top-p) control diversity, not intelligence — set them per use case, not globally.
1. From Text to Tokens: The Unit Everything Depends On
An LLM does not read characters or words; it reads tokens — sub-word units produced by a tokenizer (typically BPE or SentencePiece). Roughly, one token ≈ 4 characters ≈ ¾ of an English word, but the ratio is lossy: code, URLs, unusual formatting and non-English text all tokenise worse, sometimes two to three times worse. "Hello, world!" might be four tokens; a base64 blob or a JSON payload is far more than its word count suggests.
Why an architect cares: every commercial API prices per million tokens (split into input and output rates, output usually 2–5× more expensive), every context window is measured in tokens, and every latency estimate starts with token counts. When I size a system I always measure real prompts through a tokenizer — never estimate from word counts — because a "500-word prompt" with embedded JSON can be 40% more tokens than it looks. If you want a quick sanity check on a block of text, our word counter gets you the word side of the ratio.
Each token becomes an embedding: a high-dimensional vector (dimensions in the thousands) that positions it in a learned space where semantic similarity is geometric proximity. The same machinery — embeddings — is what powers semantic search and RAG (Part 5), which is why the two topics rhyme so strongly.
2. Self-Attention: The Intuition, Minus the Maths
Here is the only mental model of attention an architect needs. For every token the model produces a representation, it looks back at every earlier token and asks how much each should inform it. When it sees "it" in "the bank charged the fee it owed", attention learns to weight "fee" and "bank" heavily and "The" almost not at all. Those weights are learned during training — not rules a programmer wrote.
Two properties fall out of that mechanism and matter enormously:
It is O(n²) in sequence length. Every token attends to every earlier token. Double the context and the attention work quadruples — which is why a 128K-token prompt is not merely "a bit more expensive" than a 2K one, it is fundamentally a different workload. This is also why context windows are finite: the compute and memory grow quadratically even when the model could theoretically handle more.
It is why "lost in the middle" happens. Attention is not uniform; models demonstrably weight the start and end of a long context more than the middle. Burying a critical fact in paragraph 47 of a 60-page context is a real, measured failure mode — not a prompt-engineering myth. Architecturally this argues for retrieval that brings small, relevant context rather than maximal context (Part 6 is largely about this).
3. The Three Stages Behind a Model You Can Buy
When a vendor hands you "a model", they are handing you the output of a pipeline you should understand because it determines what the model is good at:
Stage 1 — Pre-training. Next-token prediction over trillions of tokens of text and code. This is where the model absorbs language, facts and reasoning patterns. It costs tens of millions of dollars and months of GPU time — which is why essentially nobody builds foundation models in-house, and why "should we train our own model" is almost always the wrong question (Part 11 covers what to do instead).
Stage 2 — Post-training / SFT. Supervised fine-tuning on instruction-following examples turns a text-completer into an assistant that answers questions, follows formats and refuses requests. This stage is comparatively cheap — days, not months — and it is the layer enterprises can actually change via fine-tuning.
Stage 3 — Alignment. RLHF or DPO shapes behaviour toward helpful, honest and harmless. This is why the model hedges, refuses and has a house personality. It is also why two models with similar benchmark scores can feel wildly different in production — alignment is taste, not capability.
The architect's takeaway: capability is mostly baked in stage 1; behaviour and format are shaped in stages 2–3. If your problem is "the model does not know our domain", fine-tuning is usually the wrong fix (use RAG); if it is "the model will not emit our exact format", fine-tuning is often the right one.
4. Inference Economics: Prefill, Decode, and the KV Cache
Running inference splits into two phases with completely different cost profiles — and confusing them is the single most common costing error I see.
Prefill processes your entire input prompt in parallel through the model — one forward pass over N tokens. It is compute-bound and produces the first output token's logits. Its duration sets TTFT (time to first token): the delay before the user sees anything. Long prompt = long TTFT, and nothing (streaming, fast decode) can hide it.
Decode then generates output one token at a time — each new token requires a full forward pass that reuses the previous computation via the KV cache (stored key/value tensors for every prior token). Decode sets TPOT (time per output token) and is memory-bandwidth-bound: the bottleneck is reading the KV cache and weights, not arithmetic. This is why a 5,000-token answer takes so much longer than a 500-token one even on the same prompt.
The KV cache is the quiet star of the economics story: it grows linearly with context length and with concurrent requests, and it lives in scarce GPU memory. A long-context, high-concurrency workload can exhaust VRAM on KV cache before it runs out of compute — which is exactly the problem PagedAttention (in vLLM) and prefix caching solve, and why Part 16 on serving exists.
Concretely for architecture: a long-prompt/short-answer workload is prefill-bound; a short-prompt/long-answer workload is decode-bound. They optimise differently — the first wants prompt compression and caching, the second wants batching and a fast decode path.
5. Sampling, Reasoning and the Smaller Knobs
A model outputs a probability distribution over the next token; sampling parameters decide how to pick from it:
- Temperature scales the distribution. 0 ≈ greedy/deterministic (for extraction, code, classification — my default for production). Higher values diversify — for creative and brainstorming workloads.
- top-p (nucleus) samples only from the smallest token set covering p of the probability mass — a softer diversity control, usually paired with temperature.
- Max tokens is a cost and latency cap — set it deliberately, it is not a quality knob.
Two important consequences. First, sampling controls diversity, not intelligence — turning temperature down will not make a wrong model right. Second, reasoning models changed the cost model: they spend extra "thinking" tokens (test-time compute) before answering, trading latency and cost for accuracy on hard problems. Architecturally that means reasoning models are a different budget line — you route to them selectively (Part 3) and cap their thinking budget, because an uncapped reasoning call is a surprise invoice.
Three more knobs worth knowing exist at the model level: Mixture-of-Experts (MoE) models activate only a fraction of parameters per token — near-frontier quality at a fraction of the inference cost, which is why so many open-weight models are MoE now. Multimodality — images, audio, video as tokens — is no longer exotic; assume new use cases will ask for it. And structured output / function calling is not a prompt trick, it is a decoding constraint that makes tool-use reliable (Part 8 builds on it).
6. What This Means for Your Architecture
Pulling it together into decisions:
- Measure tokens, not words. Instrument your prompts with real token counts before you cost anything.
- Small context beats big context. Retrieve the right 8K rather than stuffing 128K — better accuracy (lost-in-the-middle), lower prefill cost, lower latency.
- Design for the phase you are bound by. Summarisation of long docs = prefill-bound; chatbots = decode-bound; they want different hardware and caching.
- Cache aggressively. Prefix/prompt caching for repeated system prompts, semantic caching for repeated questions (Part 16).
- Set temperature per endpoint, not per app. Extraction at 0, drafting at 0.3–0.7, ideation higher — and pin it in config, not in prompts.
- Budget reasoning models. Route hard queries to them; cap thinking tokens; never let them answer what a cheap model can.
In my experience the architects who get burned are the ones who treated "the model" as a black box with a per-token price. The ones who do well understand that tokens, attention, and the KV cache create a cost and latency shape — and they design the system around that shape rather than discovering it on the first invoice.
Interview Corner
Q: Explain the difference between TTFT and TPOT and why it matters architecturally.
Strong answer: "TTFT is set by prefill — one parallel pass over the input — so it scales with prompt length and is compute-bound; it is the delay before the user sees anything and cannot be hidden by streaming. TPOT is per-output-token decode, memory-bandwidth-bound, and is what users perceive as speed once output starts. Architecturally: long-prompt workloads optimise TTFT via prompt compression and prefix caching; chatty workloads optimise TPOT via continuous batching and fast decode. You buy different hardware and cache different things for each."
What the interviewer is really testing: whether you actually understand serving economics or just quote latency averages.
Q: Why is the KV cache the real constraint on long-context serving?
Strong answer: "The KV cache stores key/value tensors for every prior token so decode does not recompute them — it trades memory for speed. It grows linearly with context length and concurrency, and it lives in scarce VRAM. So a high-concurrency, long-context workload runs out of GPU memory before it runs out of FLOPs — which is why vLLM's PagedAttention and prefix caching exist and why you cannot just 'add more context' cheaply."
What the interviewer is really testing: do you understand memory-bound inference — the thing that actually limits production scale.
Q: A stakeholder asks "can we just use a bigger context window instead of RAG?" — your answer?
Strong answer: "Rarely, and getting rarer. Three reasons: cost — stuffing 200K tokens into every query is O(n²) prefill you pay for every call, versus retrieving 8K of relevant context; accuracy — the lost-in-the-middle effect means buried facts get ignored anyway; and freshness — the context window is frozen at request time while a retrieval index updates continuously. RAG wins on all three for knowledge-heavy tasks. Big context still wins for genuinely one-shot, full-document reasoning."
What the interviewer is really testing: whether you have an opinionated default and the failure-mode reasoning to back it.
Q: When would you use a reasoning model versus a standard one?
Strong answer: "Reasoning models spend test-time compute — extra thinking tokens — for accuracy on multi-step problems: maths, code, planning, hard analysis. I route to them selectively for genuinely hard tasks and cap their thinking budget, because they are dramatically more expensive per call and slower. For extraction, classification, simple Q&A, a standard model is faster, cheaper and usually equally right. The architecture question is the router: you do not send everything to the most expensive option."
What the interviewer is really testing: cost-aware model routing, not just awareness that reasoning models exist.
Q: How do you reduce inference cost on an existing LLM feature without hurting quality?
Strong answer: "In order of effort: shrink the prompt (drop boilerplate, compress retrieved context); add caching (prompt-level and semantic); right-size the model (route easy traffic to a small model — often 70%+ of queries); batch non-interactive work; quantise if self-hosted. Only after those: consider fine-tuning a smaller model to replace the big one for the narrow task. The last resort is re-architecting the task itself to need fewer tokens."
What the interviewer is really testing: a structured cost playbook, ordered by leverage — the Part 16 discipline.
Key Takeaways
- Tokens are the unit of cost, latency and context — measure them, never estimate from words.
- Attention is O(n²) and "lost in the middle" is real — prefer small, retrieved context over maximal context.
- TTFT (prefill) and TPOT (decode) have different causes, fixes and hardware implications.
- The KV cache makes long-context serving a memory problem first, a compute problem second.
- Temperature controls diversity, not intelligence; reasoning models are a budget line, not a default.
- You buy the product of three training stages — you can only cheaply change the last two.
Frequently Asked Questions
Do I need to understand the attention maths to be a principal AI architect?
You need the intuition and the cost shape — O(n²), the KV cache, prefill vs decode — far more than the matrix algebra. You should be able to whiteboard why attention scales quadratically; you do not need to derive the QKTV softmax by hand.
How many tokens is a typical enterprise prompt?
It varies enormously, but as a rough guide: a chatbot turn might be a few hundred tokens; a RAG call with retrieved context is often 2–8K; a full-document analysis can reach tens of thousands. Always measure real prompts with a tokenizer — estimates from word counts are consistently low for structured content.
Should we fine-tune our own model instead of using an API?
Almost never as a first move — that is a Part 11 decision. Fine-tuning changes format and style, not knowledge; it makes sense for latency, cost-at-scale, or strict output contracts, after you have exhausted prompting and RAG.
Are bigger context windows making RAG obsolete?
No — they are making bad RAG obsolete. Retrieval still wins on cost, freshness and precision; large context helps for whole-document reasoning. The architectures that win use both: retrieval to find it, generous context to reason over it.
What is the one metric to watch on an LLM feature?
Cost per successful task — not cost per token, and not latency alone. A cheaper model that fails 10% of the time and needs retries is more expensive than it looks.
Further Reading
- Attention Is All You Need — Vaswani et al., the transformer paper everything builds on.
- PagedAttention / vLLM — the paper behind efficient KV-cache management and continuous batching.
- Hugging Face: How LLMs Generate Text — a clear, practical tour of decoding.
- Lost in the Middle — Liu et al. on how models use (and ignore) long context.
- OpenAI token documentation — for grounding token counts in real prompts.
