A Vocabulary for Local Coding Agents

The Terms You Need Before “Just Run It Locally” Makes Sense

Most introductions to local AI start with a download command.

That sounds like enough.

You install a runtime. You pull a model. You type a question. Text comes back. The mental model stays simple: there is a program, and the program answers.

A coding agent is not that program.

It is a loop. The model reads a prompt, may call tools, reads the tool results, and continues until it stops. Every word in that sentence hides a concept that, if unnamed, turns the first real session into a sequence of mysterious failures.

This article is a learning map for those concepts. It does not replace a model card, and it does not tell you which checkpoint to download first. It gives you the language to read the other article in this series — the one about VRAM budgets, pinned context variants, and requests that die during prefill — without treating every term as jargon.

If a sentence later says “the 27B Q4 does not fit 16 GB once KV is reserved,” you should be able to unpack it.

What an LLM Actually Produces

A large language model does not retrieve a stored answer.

It predicts the next token, then the next, given everything that came before.

A token is not a word. It is a piece of text the model was trained to count: a word, part of a word, a punctuation mark, a space, sometimes a whole short word. “Context window” of 32k means about 32,000 of those pieces, not 32,000 English words.

Rough intuition:

1 token  ≈ 4 characters of English
1000 tokens ≈ 750 words

Code and German often use more tokens than the same idea in plain English. That is why a “short” stack trace can fill a surprising amount of the window.

The model never sees your files until some layer puts their text into that window.

Prompt, Completion, and Context Window

Three words get mixed constantly.

The prompt is everything the model reads before it writes: system instructions, chat history, tool schemas, file excerpts, your latest message.

The completion is what it writes next.

The context window is the hard cap on prompt plus completion. If the window is 32,768 tokens and the prompt already uses 30,000, the model has about 2,768 tokens left to think and answer. If the prompt is larger than the window, the runtime truncates or rejects the request.

context window
    ├── prompt (input)
    └── completion (output)

A model card that says “256k native context” means the architecture can attend over that many tokens. It does not mean your GPU can hold the memory for that many tokens. Those are different facts.

Parameters, Dense Models, and MoE

Parameters are the learned numbers in the network. People say “7B” or “27B” for seven or twenty-seven billion of them.

More parameters usually means more capacity and more memory.

A dense model uses (almost) all parameters for every token. Qwen3.8 27B is dense. Every generated token pays the full 27B cost.

A Mixture of Experts (MoE) model has many parameters on disk, but only a subset is active per token. GPT-OSS 20B and LFM2.5 8B-A1B are in that family. “8B-A1B” means about eight billion total, about one billion active. Disk size still follows the total. Speed often follows the active set.

dense 27B     → large file, large compute every token
MoE 20B / 3B  → large file, cheaper compute per token

That is why a 20B MoE can feel faster than a 14B dense model, and why “it is only 20B” is not a VRAM guarantee.

Quantization: Why Q4 and Q8 Are Not Different Models

The training checkpoint is often stored in 16-bit floats. That is accurate and huge.

Quantization stores the same network with fewer bits per number. Q4_K_M is a common 4-bit recipe. Q8_0 is 8-bit. The name of the model stays. The file shrinks.

same architecture
same tokenizer
different number format
    Q4  → smaller, faster, slightly less precise
    Q8  → larger, closer to the original
    BF16 → full-ish precision, often will not fit

On a 16 GB card, a 9B model at Q4 may leave room for a long context. The same 9B at Q8 uses more of the card for weights and less for history. The Hugging Face parameter filter does not see that difference. It counts billions of parameters, not gigabytes of VRAM.

RAM, VRAM, GPU, and Offload

RAM is system memory. The CPU uses it.

VRAM is memory on the graphics card. The GPU uses it.

Local inference wants the model weights and the KV cache in VRAM. If they do not fit, the runtime offloads layers to RAM and runs them on the CPU.

That still “works.” It is much slower.

The check is not the download size. The check is the process list:

ollama ps

You want 100% GPU. Any CPU percentage means part of the network is commuting through system RAM.

A useful first equation:

VRAM ≈ weights + vision projector + KV cache + runtime overhead

Weights are almost fixed per quant. The KV cache grows with the context window.

KV Cache: Why Long Context Costs Memory Twice

When the model reads a token, it computes keys and values for attention and keeps them.

That store is the KV cache.

Without it, every new token would re-read the entire prompt from scratch. With it, generation can reuse what was already computed.

The cost is memory that grows with:

number of layers
number of KV heads
hidden size
context length

So a “128k model” is not only a smarter window. It is a reservation of cache. Pinning num_ctx to 32,768 instead of 131,072 does not change the downloaded weights. It changes how much cache the runtime allocates.

That is why one 27B file can appear as several tags in a picker: 32k, 64k, 96k, 128k. They are configurations, not four downloads.

Prefill and Decode: The Two Halves of One Request

A completion has two phases.

Prefill reads the prompt and builds the KV cache. No visible answer yet. On a local GPU this can take tens of seconds for a 40k-token prompt, even at hundreds of tokens per second.

Decode writes the completion, one token after another. This is the streaming text you see.

request
  1. prefill   ← silent, expensive, cache-building
  2. decode    ← visible tokens

Cloud clients often assume that silence means the connection died. Local prefill is silent by nature. If the HTTP client cancels after eighteen seconds, the runtime log shows a 500, cancel task, and truncated = 0. Nothing was generated. The prompt was still being read.

Prefix cache or context checkpoint is a related idea. If the next request starts with the same tokens as the last one, the runtime can restore part of the KV cache. If the conversation branched, it restores only the shared prefix and prefills the rest.

That is why a long session suddenly feels slow after you change the system prompt, switch models, or compress history badly: the cheap prefix is gone.

Ollama, Tags, and the OpenAI-Compatible Surface

Ollama is a local runtime. It downloads a packaged model, keeps it loaded, and serves HTTP.

A tag is a name plus an optional variant: qwen3.8:latest, granite4.1:8b. Your own tag, created with a Modelfile, is just another name pointing at the same blobs plus different parameters.

The API most agents speak is OpenAI-compatible:

POST http://127.0.0.1:11434/v1/chat/completions

The agent does not need to know Ollama’s native format. It needs a base URL, a model id, and usually a dummy API key because the client library requires one.

If the id in the agent config does not match the Ollama tag, the request hits the wrong model or no model. That bug looks like “the picker is empty” or “context is 4k again.”

What a Coding Agent Adds on Top of a Chat Model

A chat model answers in text.

A coding agent is a program that:

sends a prompt
receives either text or a tool call
executes the tool
appends the result
repeats

That cycle is the tool loop.

Tools are functions the model is allowed to request: read a file, edit a file, run a shell command, search the repo. The model does not execute them. The agent host does.

The system prompt tells the model who it is and how to use those tools.

Memory files and skills are extra text loaded every turn. They are useful, and they are not free. They occupy the context window before you type.

A typical overhead on a real agent is already in the mid-teens of thousands of tokens. That is why a 16k window is not a “fast profile.” It is a window that is already full.

Thinking or reasoning tokens are an intermediate trace some models produce before the visible answer. They count against max_tokens and against the window. A thinking model can “stop” because it spent the output budget on the trace.

Native Context Versus Configured Context

Two numbers appear on every local setup. They are not synonyms.

Native context is what the architecture was trained or extended to support. Qwen3.5 9.7B natively goes to 256k. Qwen2.5 Coder 14B natively stops at 32k. Declaring 128k in the client for a 32k-native model is a lie. The runtime cannot invent trained length.

Configured context (num_ctx) is what you reserved this time. You almost always set it lower than native so the KV cache fits VRAM.

native     = theoretical maximum
num_ctx    = what this variant actually allocated
client     = what the agent believes the window is

All three must agree. If they do not, you get silent truncation, fake long context, or a GPU that starts offloading.

Why the Agent “Stops”

Learners often treat every pause as one bug. There are at least four.

Approval. The host asks you before a shell command or an edit. The model is waiting on you, not thinking.

Turn cap. The client has a maximum number of model/tool rounds. When it hits the cap, the run ends even if the task is unfinished.

Yield. The model writes a paragraph instead of the next tool call. Small local models do this often. The runtime is fine. The policy in the model’s head was “I am done.”

Cancelled prefill. The HTTP request dies before decode. The log shows progress through prompt processing, then cancel task. The fix is timeouts that tolerate silence, plus shorter prompts.

If you cannot name which of the four happened, you will keep restarting the same session.

The Shell Is a Tool, Not “Your Terminal”

The agent’s shell is a subprocess with a specific executable.

On Windows that is often cmd.exe, chosen from ComSpec. It does not have head, grep, or Select-String. Those belong to Unix and to PowerShell.

The German error:

Der Befehl "..." ist entweder falsch geschrieben oder konnte nicht gefunden werden.

is cmd.exe telling you the command is not on its PATH.

This is not a model stupidity problem first. It is an environment invariant. The model must be told which language the tool speaks, or every build becomes a retry loop.

A Minimal Mental Model

You can hold the whole stack in one diagram:

you
  → coding agent (tools, memory, approvals, timeouts)
    → OpenAI-compatible HTTP
      → Ollama
        → model weights (quantized)
        → KV cache (sized by num_ctx)
        → GPU if it fits, else CPU offload

Each arrow can fail independently.

The model can be excellent and the request still dies in prefill.

The GPU can be empty and the agent still stop because it asked for confirmation.

The context window can be huge and still be useless if the first 17k tokens are overhead.

Once those sentences are obvious, the rest is configuration.

How to Practice the Vocabulary

Do not start by collecting twenty models.

Start by measuring one.

ollama ps

Load a small model. Send a short prompt. Note GPU percent.

Raise num_ctx. Watch whether GPU percent drops.

Send a long prompt. Notice the delay before the first token. That delay is prefill.

Open the agent. Run /context if it exists. Read how much of the window is already gone before you speak.

Then read a model card again. “27B”, “Q4_K_M”, “256k”, “tools”, “thinking”, “vision” should now each point to a concrete cost.

Final Takeaway

You do not need to become a researcher to run a local coding agent.

You need names for the parts that spend memory, time, and tokens.

A token is the unit of context. Parameters and quantization decide the weight file. VRAM holds weights plus KV cache. Prefill is the silent half of the request. The agent is a tool loop around a model, not the model itself. Offload, approval, turn caps, and cancelled prefills are different ways the loop stops.

Once those words are stable, “just run it locally” becomes a set of decisions instead of a hope.

The companion piece in this series applies that vocabulary to a 16 GB workstation. This piece exists so that article does not have to teach the dictionary and the architecture at the same time.