{"id":222,"date":"2026-08-23T13:27:51","date_gmt":"2026-08-23T13:27:51","guid":{"rendered":"https:\/\/www.fabricioruch.ch\/?p=222"},"modified":"2026-08-23T13:27:51","modified_gmt":"2026-08-23T13:27:51","slug":"the-coding-agent-did-not-stall-the-runtime-did","status":"publish","type":"post","link":"https:\/\/www.fabricioruch.ch\/?p=222","title":{"rendered":"The Coding Agent Did Not Stall. The Runtime Did."},"content":{"rendered":"\n

Running Qwen Code on Ollama Is an Architecture Problem, Not a Prompt Problem<\/h2>\n\n\n\n

Most local coding setups start with a simple assumption: pull a strong model, point the agent at localhost:11434<\/code>, and work as if the cloud never existed.<\/p>\n\n\n\n

That sounds clean.<\/p>\n\n\n\n

The model is on disk. The GPU is in the machine. The agent has tools. Tokens are free. Latency should be the only remaining concern.<\/p>\n\n\n\n

Then the first real session arrives.<\/p>\n\n\n\n

A 27B model fills 16 GB of VRAM before the conversation even starts. A 128k context window looks available in the picker and then offloads to CPU. The agent asks for confirmation after every shell command. An edit fails because the file was never read in this session. A request sits silent for eighteen seconds, returns HTTP 500, and the slot goes idle.<\/p>\n\n\n\n

Nothing in that list is a prompting failure.<\/p>\n\n\n\n

The local agent stack is a small distributed system. The model, the KV cache, the OpenAI-compatible proxy, the tool loop, and the desktop shell all have different failure modes. If those boundaries are left implicit, the session feels broken. If they are named, the setup becomes predictable.<\/p>\n\n\n\n

This article is a walkthrough of that system as I run it on a workstation with 16 GB of VRAM: Ollama as the runtime, Qwen Code as the agent, and a set of pinned context variants instead of one \u201clargest possible\u201d model.<\/p>\n\n\n\n

\n\n\n\n

The Cloud Assumption Does Not Survive Local Hardware<\/h2>\n\n\n\n

A hosted coding agent hides three expensive facts.<\/p>\n\n\n\n

First, the model already fits. The provider sized the GPU.<\/p>\n\n\n\n

Second, the context window is a product feature, not a memory budget. 128k means 128k.<\/p>\n\n\n\n

Third, the client can wait. Prefill, thinking tokens, and tool rounds are someone else\u2019s capacity problem.<\/p>\n\n\n\n

Local inference inverts all three.<\/p>\n\n\n\n

On a 16 GB card, a Q4 27B checkpoint is already about 17 GB of weights. The vision projector, the KV cache, and the runtime overhead still have to land somewhere. Ollama will run the model anyway. It will just put layers on the CPU. ollama ps<\/code> then shows something like 61% CPU \/ 39% GPU<\/code>, and the session becomes slow enough that the client gives up.<\/p>\n\n\n\n

The useful question is not \u201cwhich model is best.\u201d<\/p>\n\n\n\n

The useful question is:<\/p>\n\n\n\n

What still stays at 100% GPU\nfor the context length this agent actually needs?<\/code><\/pre>\n\n\n\n
That is an architecture question. It decides which checkpoint you load, which num_ctx<\/code> you pin, and which client timeouts you disable.<\/p>\n\n\n\n
\n\n\n\n
VRAM Is Weights Plus Cache, Not a Model Name<\/h2>\n\n\n\n
A model card that says \u201c256k context\u201d describes capability, not a safe operating point.<\/p>\n\n\n\n
The working set is roughly:<\/p>\n\n\n\n
VRAM \u2248 weights + projector + KV(cache, num_ctx) + runtime<\/code><\/pre>\n\n\n\n
The weights are fixed per quant. The KV cache grows with context. A 9.7B Q4 checkpoint around 6.6 GB can keep a long window on a 16 GB card. A 27B Q4 checkpoint around 17 GB cannot, even at 16k.<\/p>\n\n\n\n
That produces a simple policy:<\/p>\n\n\n\n
small dense \/ small MoE  \u2192 long context, 100% GPU\n27B-class dense          \u2192 short or medium context, expect offload\n35B MoE \/ 30B agent      \u2192 treat 128k as a luxury, not a default<\/code><\/pre>\n\n\n\n
On this machine the practical map looks like this:<\/p>\n\n\n\n
Qwen3.5 9.7B   ~6.6 GB   192k\u2013224k still GPU-safe\nLFM2.5 8B-A1B  ~5.2 GB   128k GPU-safe\nGranite 4.1 8B ~5.3 GB   128k GPU-safe\nGPT-OSS 20B    ~13 GB    64k is the GPU default\nQwen3.8 27B    ~17 GB    offload even at 16k\nMuse Glimmer   ~18 GB    offload is unavoidable<\/code><\/pre>\n\n\n\n
The 27B Qwen3.8 build is a real model. It is not a 16 GB model.<\/p>\n\n\n\n
Hugging Face search does not change that. A filter for 6B\u20139B plus the string qwen3.8<\/code> does not find an official 9B Qwen3.8. The official 3.8 line currently publishes a 27B dense model and a frontier MoE. The 9B hits are community distillations of 3.8 behavior into the Qwen3.5-9B architecture. That can be useful. It is not the same artifact as Qwen\/Qwen3.8-27B<\/code>.<\/p>\n\n\n\n
\n\n\n\n
Context Variants Are Configuration, Not New Models<\/h2>\n\n\n\n
Ollama makes it cheap to lie to yourself.<\/p>\n\n\n\n
ollama create<\/code> with a Modelfile does not download another 17 GB. It pins a parameter:<\/p>\n\n\n\n
FROM qwen3.8:latest\nPARAMETER num_ctx 32768<\/code><\/pre>\n\n\n\n
The weights stay the same. Only the reserved KV window changes.<\/p>\n\n\n\n
That is the correct way to expose a family in the agent picker:<\/p>\n\n\n\n
qwen3.8-qwen-32k\nqwen3.8-qwen-64k\nqwen3.8-qwen-96k\nqwen3.8-qwen<\/code><\/pre>\n\n\n\n
The last one is 128k. It exists because the architecture supports it. It is not the daily driver on 16 GB.<\/p>\n\n\n\n
The same pattern applies to GPT-OSS 20B (64k \/ 96k \/ 128k) and to the small GPU-safe models (32k \/ 64k \/ 96k \/ 128k).<\/p>\n\n\n\n
Two details matter for the client.<\/p>\n\n\n\n
The provider id<\/code> must match the Ollama tag exactly. A mismatch fails silently or selects the wrong window.<\/p>\n\n\n\n
The declared contextWindowSize<\/code> must match num_ctx<\/code>. If the client believes the window is 128k while the runtime reserved 32k, the agent will stuff the prompt until the runtime truncates or the request dies.<\/p>\n\n\n\n
Qwen Code version 4 has a sharper trap. modelProviders.openai<\/code> must be a bare array. A wrapped object with protocol<\/code> and models<\/code> is skipped. The picker then looks configured and is empty.<\/p>\n\n\n\n
\n\n\n\n
Sixteen Thousand Tokens Is Not an Agent Window<\/h2>\n\n\n\n
A coding agent does not start at token zero.<\/p>\n\n\n\n
Before the user types anything, the runtime already holds:<\/p>\n\n\n\n
system prompt\nbuilt-in tool schemas\nmemory files\nskills<\/code><\/pre>\n\n\n\n
On this setup that overhead is roughly 17k tokens.<\/p>\n\n\n\n
A 16k variant is therefore not a \u201cfast GPU profile.\u201d It is a window that is already full. Qwen Code warns about it. The model then has no room to call tools without evicting the instructions that make tool calling work.<\/p>\n\n\n\n
The practical minimum for this agent is 32k. 64k is the first window that feels like a working session.<\/p>\n\n\n\n
That is another reason not to chase native maxima. A 256k card on a 9.7B model is usable. A 16k card on a 27B model is not an optimization. It is a compression of the agent itself.<\/p>\n\n\n\n
\n\n\n\n
The Agent Stops for Four Different Reasons<\/h2>\n\n\n\n
\u201cThe model keeps stopping\u201d is not one bug.<\/p>\n\n\n\n
It is four different layers, and they look identical in the chat UI: the turn ends, and you have to nudge it.<\/p>\n\n\n\n
1. The client is waiting for approval<\/h3>\n\n\n\n
The default approval mode still interrupts on risky shell commands. auto<\/code> still asks. yolo<\/code> does not.<\/p>\n\n\n\n
That is a policy choice, not a model choice. On a local workstation I accept the risk and keep the loop moving. On an untrusted repo I would not.<\/p>\n\n\n\n
2. The session has a turn cap<\/h3>\n\n\n\n
model.maxSessionTurns<\/code> is a client budget. When it hits the cap, the run ends even if the task is unfinished. -1<\/code> removes that cap.<\/p>\n\n\n\n
3. The model yields instead of calling the next tool<\/h3>\n\n\n\n
Small local models often write a status paragraph after two or three tool calls. The runtime did not crash. The model decided the turn was over.<\/p>\n\n\n\n
A persistent instruction helps, but it does not replace model quality. Long agent loops want a stronger tool-using checkpoint, not a shorter prompt.<\/p>\n\n\n\n
4. The HTTP request dies during prefill<\/h3>\n\n\n\n
This one looks like a model hang. The Ollama log says something else.<\/p>\n\n\n\n
n_ctx_slot = 131072\ntask.n_tokens = 42003\nrestored context checkpoint ... n_tokens = 25846\nprompt processing ... 680 tokens per second\n[GIN] 500 | 18.2s | POST \/v1\/chat\/completions\nsrv stop: cancel task\nslot release ... truncated = 0<\/code><\/pre>\n\n\n\n
Read that sequence carefully.<\/p>\n\n\n\n
The prompt is 42k tokens. The prefix cache only matches to 25k. The remaining 16k is being prefilled at a healthy GPU rate. No completion token has been produced. Then the HTTP client cancels. Ollama records 500, releases the slot, and goes idle.<\/p>\n\n\n\n
The agent did not finish thinking.<\/p>\n\n\n\n
The client stopped waiting for the first byte.<\/p>\n\n\n\n
Qwen Code\u2019s stream guards are designed for cloud APIs that start dripping tokens quickly. Local prefill is silent. A 42k prompt at 680 tokens per second needs about a minute if the cache is cold, and about twenty seconds if 25k is already cached. An 18 second abort is shorter than that.<\/p>\n\n\n\n
The fix is not a better system prompt. The fix is to stop treating silence as death:<\/p>\n\n\n\n
QWEN_CODE_API_TIMEOUT_MS=0\nQWEN_STREAM_IDLE_TIMEOUT_MS=0\nQWEN_STREAM_MAX_LIFETIME_MS=0<\/code><\/pre>\n\n\n\n
And then stop sending 42k prompts as if they were free. \/compress<\/code> is not a convenience command. It is how you keep the next prefill inside a window the client will actually wait for.<\/p>\n\n\n\n
\n\n\n\n
The Desktop Shell Is Part of the Architecture<\/h2>\n\n\n\n
The agent\u2019s shell tool is not \u201cyour terminal.\u201d<\/p>\n\n\n\n
On Windows, Qwen Code selects the executable from ComSpec<\/code>. The default is cmd.exe<\/code>. That is why this fails:<\/p>\n\n\n\n
git show c06d6a1 | head -300<\/code><\/pre>\n\n\n\n
and why this also fails:<\/p>\n\n\n\n
dotnet build 2>&1 | Select-String error<\/code><\/pre>\n\n\n\n
head<\/code> is Unix. Select-String<\/code> is PowerShell. cmd.exe<\/code> knows neither.<\/p>\n\n\n\n
The error text is the giveaway:<\/p>\n\n\n\n
Der Befehl \"...\" ist entweder falsch geschrieben oder konnte nicht gefunden werden.<\/code><\/pre>\n\n\n\n
That is cmd.exe, not PowerShell.<\/p>\n\n\n\n
Qwen Code will use PowerShell if ComSpec<\/code> ends in powershell.exe<\/code> or pwsh.exe<\/code>. Setting that globally is a bad idea. Windows starts .cmd<\/code> launchers through ComSpec<\/code>. Pointing it at pwsh<\/code> hangs qwen.cmd<\/code>.<\/p>\n\n\n\n
The durable fix is a wrapper that sets ComSpec<\/code> only for the Node entrypoint, plus a hard instruction: this machine speaks cmd.exe or an explicit pwsh -NoProfile -Command<\/code>, never bash.<\/p>\n\n\n\n
This belongs in the same category as VRAM. It is an environment invariant. If the model does not know it, every build step becomes a retry loop.<\/p>\n\n\n\n
\n\n\n\n
Edits Fail for the Same Class of Reason<\/h2>\n\n\n\n
A string-replace miss is often not a mismatch.<\/p>\n\n\n\n
The tool layer rejects the edit first:<\/p>\n\n\n\n
File ... has not been read in this session.\nUse the read_file tool first ...<\/code><\/pre>\n\n\n\n
That check is not configurable. The agent must read the file in this session, then send old_string<\/code> bytes that come from that read.<\/p>\n\n\n\n
Retrying the same edit is wasted work. The second attempt is not closer to matching. It is still blocked by the prior-read rule.<\/p>\n\n\n\n
This is the same design idea as staged rule evaluation. The tool will not pretend it knows the file. unknown<\/code> here is \u201cyou have not loaded the bytes.\u201d The correct next step is to acquire them.<\/p>\n\n\n\n
\n\n\n\n
What I Keep as Daily Defaults<\/h2>\n\n\n\n
The picker has many variants. The daily set is small.<\/p>\n\n\n\n
Fast agent loop, 100% GPU\n  qwen3.5 9.7B at 192k\n  or lfm2.5 at 128k\n\nStronger tools, accept some offload\n  qwen3.8 27B at 32k or 64k\n  gpt-oss 20B at 64k\n\nVision plus tools on a small card\n  gemma4 12B at 128k<\/code><\/pre>\n\n\n\n
I do not use 16k agent profiles. I do not use 256k on the 9.7B model unless I am testing the ceiling. I do not treat a 27B 128k tag as \u201cthe good one.\u201d<\/p>\n\n\n\n
The client settings that actually keep the loop alive are not model tags:<\/p>\n\n\n\n
tools.approvalMode = yolo\nmodel.maxSessionTurns = -1\nstream idle \/ lifetime \/ API timeout = 0<\/code><\/pre>\n\n\n\n
Those settings make the system honest about local cost. They do not make a 23 GB model fit in 16 GB. They only stop the client from abandoning a request that was still prefilling correctly.<\/p>\n\n\n\n
\n\n\n\n
Final Takeaway<\/h2>\n\n\n\n
A local coding agent is not a smaller cloud agent.<\/p>\n\n\n\n
It is a runtime with a hard memory budget, a silent prefill phase, a Windows shell that is not bash, and a tool loop that will stop for policy, for tokens, for model habit, or for an HTTP cancel.<\/p>\n\n\n\n
The valuable configuration is not the largest context number on the model card.<\/p>\n\n\n\n
It is the largest window that stays on the GPU, the smallest window that still holds the agent overhead, and a client that does not mistake prefill silence for failure.<\/p>\n\n\n\n
The model can only continue if the request is still alive.<\/p>\n\n\n\n
That is not a prompt. That is the architecture.<\/p>\n","protected":false},"excerpt":{"rendered":"
Running Qwen Code on Ollama Is an Architecture Problem, Not a Prompt Problem Most local coding setups start with a simple assumption: pull a strong model, point the agent at localhost:11434, and work as if the cloud never existed. That sounds clean. The model is on disk. The GPU is in the machine. The agent […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[38,37,40],"tags":[],"class_list":["post-222","post","type-post","status-publish","format-standard","hentry","category-ai-automation","category-ai-engineering","category-llm-orchestration"],"_links":{"self":[{"href":"https:\/\/www.fabricioruch.ch\/index.php?rest_route=\/wp\/v2\/posts\/222","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fabricioruch.ch\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fabricioruch.ch\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fabricioruch.ch\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fabricioruch.ch\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=222"}],"version-history":[{"count":1,"href":"https:\/\/www.fabricioruch.ch\/index.php?rest_route=\/wp\/v2\/posts\/222\/revisions"}],"predecessor-version":[{"id":224,"href":"https:\/\/www.fabricioruch.ch\/index.php?rest_route=\/wp\/v2\/posts\/222\/revisions\/224"}],"wp:attachment":[{"href":"https:\/\/www.fabricioruch.ch\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=222"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fabricioruch.ch\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=222"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fabricioruch.ch\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=222"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}