The Coding Agent Did Not Stall. The Runtime Did.

Running Qwen Code on Ollama Is an Architecture Problem, Not a Prompt Problem

Most local coding setups start with a simple assumption: pull a strong model, point the agent at localhost:11434, and work as if the cloud never existed.

That sounds clean.

The model is on disk. The GPU is in the machine. The agent has tools. Tokens are free. Latency should be the only remaining concern.

Then the first real session arrives.

A 27B model fills 16 GB of VRAM before the conversation even starts. A 128k context window looks available in the picker and then offloads to CPU. The agent asks for confirmation after every shell command. An edit fails because the file was never read in this session. A request sits silent for eighteen seconds, returns HTTP 500, and the slot goes idle.

Nothing in that list is a prompting failure.

The local agent stack is a small distributed system. The model, the KV cache, the OpenAI-compatible proxy, the tool loop, and the desktop shell all have different failure modes. If those boundaries are left implicit, the session feels broken. If they are named, the setup becomes predictable.

This article is a walkthrough of that system as I run it on a workstation with 16 GB of VRAM: Ollama as the runtime, Qwen Code as the agent, and a set of pinned context variants instead of one “largest possible” model.

The Cloud Assumption Does Not Survive Local Hardware

A hosted coding agent hides three expensive facts.

First, the model already fits. The provider sized the GPU.

Second, the context window is a product feature, not a memory budget. 128k means 128k.

Third, the client can wait. Prefill, thinking tokens, and tool rounds are someone else’s capacity problem.

Local inference inverts all three.

On a 16 GB card, a Q4 27B checkpoint is already about 17 GB of weights. The vision projector, the KV cache, and the runtime overhead still have to land somewhere. Ollama will run the model anyway. It will just put layers on the CPU. ollama ps then shows something like 61% CPU / 39% GPU, and the session becomes slow enough that the client gives up.

The useful question is not “which model is best.”

The useful question is:

What still stays at 100% GPU
for the context length this agent actually needs?

That is an architecture question. It decides which checkpoint you load, which num_ctx you pin, and which client timeouts you disable.

VRAM Is Weights Plus Cache, Not a Model Name

A model card that says “256k context” describes capability, not a safe operating point.

The working set is roughly:

VRAM ≈ weights + projector + KV(cache, num_ctx) + runtime

The weights are fixed per quant. The KV cache grows with context. A 9.7B Q4 checkpoint around 6.6 GB can keep a long window on a 16 GB card. A 27B Q4 checkpoint around 17 GB cannot, even at 16k.

That produces a simple policy:

small dense / small MoE  → long context, 100% GPU
27B-class dense          → short or medium context, expect offload
35B MoE / 30B agent      → treat 128k as a luxury, not a default

On this machine the practical map looks like this:

Qwen3.5 9.7B   ~6.6 GB   192k–224k still GPU-safe
LFM2.5 8B-A1B  ~5.2 GB   128k GPU-safe
Granite 4.1 8B ~5.3 GB   128k GPU-safe
GPT-OSS 20B    ~13 GB    64k is the GPU default
Qwen3.8 27B    ~17 GB    offload even at 16k
Muse Glimmer   ~18 GB    offload is unavoidable

The 27B Qwen3.8 build is a real model. It is not a 16 GB model.

Hugging Face search does not change that. A filter for 6B–9B plus the string qwen3.8 does not find an official 9B Qwen3.8. The official 3.8 line currently publishes a 27B dense model and a frontier MoE. The 9B hits are community distillations of 3.8 behavior into the Qwen3.5-9B architecture. That can be useful. It is not the same artifact as Qwen/Qwen3.8-27B.

Context Variants Are Configuration, Not New Models

Ollama makes it cheap to lie to yourself.

ollama create with a Modelfile does not download another 17 GB. It pins a parameter:

FROM qwen3.8:latest
PARAMETER num_ctx 32768

The weights stay the same. Only the reserved KV window changes.

That is the correct way to expose a family in the agent picker:

qwen3.8-qwen-32k
qwen3.8-qwen-64k
qwen3.8-qwen-96k
qwen3.8-qwen

The last one is 128k. It exists because the architecture supports it. It is not the daily driver on 16 GB.

The same pattern applies to GPT-OSS 20B (64k / 96k / 128k) and to the small GPU-safe models (32k / 64k / 96k / 128k).

Two details matter for the client.

The provider id must match the Ollama tag exactly. A mismatch fails silently or selects the wrong window.

The declared contextWindowSize must match num_ctx. If the client believes the window is 128k while the runtime reserved 32k, the agent will stuff the prompt until the runtime truncates or the request dies.

Qwen Code version 4 has a sharper trap. modelProviders.openai must be a bare array. A wrapped object with protocol and models is skipped. The picker then looks configured and is empty.

Sixteen Thousand Tokens Is Not an Agent Window

A coding agent does not start at token zero.

Before the user types anything, the runtime already holds:

system prompt
built-in tool schemas
memory files
skills

On this setup that overhead is roughly 17k tokens.

A 16k variant is therefore not a “fast GPU profile.” It is a window that is already full. Qwen Code warns about it. The model then has no room to call tools without evicting the instructions that make tool calling work.

The practical minimum for this agent is 32k. 64k is the first window that feels like a working session.

That is another reason not to chase native maxima. A 256k card on a 9.7B model is usable. A 16k card on a 27B model is not an optimization. It is a compression of the agent itself.

The Agent Stops for Four Different Reasons

“The model keeps stopping” is not one bug.

It is four different layers, and they look identical in the chat UI: the turn ends, and you have to nudge it.

1. The client is waiting for approval

The default approval mode still interrupts on risky shell commands. auto still asks. yolo does not.

That is a policy choice, not a model choice. On a local workstation I accept the risk and keep the loop moving. On an untrusted repo I would not.

2. The session has a turn cap

model.maxSessionTurns is a client budget. When it hits the cap, the run ends even if the task is unfinished. -1 removes that cap.

3. The model yields instead of calling the next tool

Small local models often write a status paragraph after two or three tool calls. The runtime did not crash. The model decided the turn was over.

A persistent instruction helps, but it does not replace model quality. Long agent loops want a stronger tool-using checkpoint, not a shorter prompt.

4. The HTTP request dies during prefill

This one looks like a model hang. The Ollama log says something else.

n_ctx_slot = 131072
task.n_tokens = 42003
restored context checkpoint ... n_tokens = 25846
prompt processing ... 680 tokens per second
[GIN] 500 | 18.2s | POST /v1/chat/completions
srv stop: cancel task
slot release ... truncated = 0

Read that sequence carefully.

The prompt is 42k tokens. The prefix cache only matches to 25k. The remaining 16k is being prefilled at a healthy GPU rate. No completion token has been produced. Then the HTTP client cancels. Ollama records 500, releases the slot, and goes idle.

The agent did not finish thinking.

The client stopped waiting for the first byte.

Qwen Code’s stream guards are designed for cloud APIs that start dripping tokens quickly. Local prefill is silent. A 42k prompt at 680 tokens per second needs about a minute if the cache is cold, and about twenty seconds if 25k is already cached. An 18 second abort is shorter than that.

The fix is not a better system prompt. The fix is to stop treating silence as death:

QWEN_CODE_API_TIMEOUT_MS=0
QWEN_STREAM_IDLE_TIMEOUT_MS=0
QWEN_STREAM_MAX_LIFETIME_MS=0

And then stop sending 42k prompts as if they were free. /compress is not a convenience command. It is how you keep the next prefill inside a window the client will actually wait for.

The Desktop Shell Is Part of the Architecture

The agent’s shell tool is not “your terminal.”

On Windows, Qwen Code selects the executable from ComSpec. The default is cmd.exe. That is why this fails:

git show c06d6a1 | head -300

and why this also fails:

dotnet build 2>&1 | Select-String error

head is Unix. Select-String is PowerShell. cmd.exe knows neither.

The error text is the giveaway:

Der Befehl "..." ist entweder falsch geschrieben oder konnte nicht gefunden werden.

That is cmd.exe, not PowerShell.

Qwen Code will use PowerShell if ComSpec ends in powershell.exe or pwsh.exe. Setting that globally is a bad idea. Windows starts .cmd launchers through ComSpec. Pointing it at pwsh hangs qwen.cmd.

The durable fix is a wrapper that sets ComSpec only for the Node entrypoint, plus a hard instruction: this machine speaks cmd.exe or an explicit pwsh -NoProfile -Command, never bash.

This belongs in the same category as VRAM. It is an environment invariant. If the model does not know it, every build step becomes a retry loop.

Edits Fail for the Same Class of Reason

A string-replace miss is often not a mismatch.

The tool layer rejects the edit first:

File ... has not been read in this session.
Use the read_file tool first ...

That check is not configurable. The agent must read the file in this session, then send old_string bytes that come from that read.

Retrying the same edit is wasted work. The second attempt is not closer to matching. It is still blocked by the prior-read rule.

This is the same design idea as staged rule evaluation. The tool will not pretend it knows the file. unknown here is “you have not loaded the bytes.” The correct next step is to acquire them.

What I Keep as Daily Defaults

The picker has many variants. The daily set is small.

Fast agent loop, 100% GPU
  qwen3.5 9.7B at 192k
  or lfm2.5 at 128k

Stronger tools, accept some offload
  qwen3.8 27B at 32k or 64k
  gpt-oss 20B at 64k

Vision plus tools on a small card
  gemma4 12B at 128k

I do not use 16k agent profiles. I do not use 256k on the 9.7B model unless I am testing the ceiling. I do not treat a 27B 128k tag as “the good one.”

The client settings that actually keep the loop alive are not model tags:

tools.approvalMode = yolo
model.maxSessionTurns = -1
stream idle / lifetime / API timeout = 0

Those settings make the system honest about local cost. They do not make a 23 GB model fit in 16 GB. They only stop the client from abandoning a request that was still prefilling correctly.

Final Takeaway

A local coding agent is not a smaller cloud agent.

It is a runtime with a hard memory budget, a silent prefill phase, a Windows shell that is not bash, and a tool loop that will stop for policy, for tokens, for model habit, or for an HTTP cancel.

The valuable configuration is not the largest context number on the model card.

It is the largest window that stays on the GPU, the smallest window that still holds the agent overhead, and a client that does not mistake prefill silence for failure.

The model can only continue if the request is still alive.

That is not a prompt. That is the architecture.