LLM for Reasoning, Scripts for Execution: What an 8B Model on a CPU Taught Me

I run a local LLM on a CPU-only server with no GPU. Two days of benchmarks changed how I build agent workflows, and it can be summed up in one line:

The LLM is the brain, not the hands. It plans, reasons and explains. Scripts execute.

Hardware: 2 × Xeon E5-2680 v3 (a 2014-era part), 125 GB RAM, no GPU. Inference with llama.cpp. (I’ve written up the setup in Running a Local LLM on CPU.)

Part 1 — Getting the model to respond at all

Model	Test	Context tokens	Time	Result
Qwen3-14B Q4_K_M (9.0 GB)	Simple “Hi”	2	~3 s	OK
Qwen3-14B Q4_K_M	RAG context	368	> 600 s	Timeout
Qwen3-14B Q4_K_M	Full context	8000	> 600 s	Timeout
Qwen3-8B Q4_K_M (4.7 GB)	Simple query	43	21.7 s	OK
Qwen3-8B Q4_K_M	RAG context	368	39.9 s	OK
Qwen3-8B Q4_K_M	Full context	8000	~3–5 min	Slow but works

Switching from 14B to 8B was the single biggest win. The model is about half the size, prompt processing (prefill) is 15–20× faster by my timings, and generation speed is similar at roughly 8–10 tokens/s. A RAG query went from “never finishes” to “about 40 seconds”.

The RAG pipeline itself was straightforward: ChromaDB with all-MiniLM-L6-v2 embeddings over 86 chunks of internal documentation. Retrieval quality was good in my spot checks (all five returned chunks were relevant). The real gain is context size: retrieval cut the prompt from about 8000 tokens to 368, roughly 95% less, and on a slow CPU, prompt tokens are what hurt.

Takeaways:

A 14B model on this CPU is unusable; even a few hundred tokens of context timed out.
An 8B model on the same box is usable, and RAG makes it practical.
The bottleneck was inference speed on old hardware, not tokenisation.
A GPU would still help enormously. My own estimate is that a used 24 GB card would bring RAG queries under 5 seconds. I haven’t tested that, so treat it as a guess and check current prices before buying.

Part 2 — Then I asked: why use the LLM at all for this?

Many of my agent’s jobs are not reasoning. They’re “fetch this list from an API”, “check this VM’s status”, “export this table”. I benchmarked the LLM against a plain script on the same tasks (a cloud model handled orchestration and planning in both cases):

Task	Script	LLM (8B + RAG)	Winner
Export a data report	2 s	61 s	Script, ~30×
List fees for a class	1 s	45–60 s	Script, ~45×
Full data query	3 s	90–120 s	Script, ~40×
Proxmox VM status	3 s	40–60 s	Script, ~15×
System health check	1 s	n/a (no context)	Script
Troubleshooting / reasoning	n/a	20–40 s	LLM, the only option
Code generation or fixes	n/a	30–90 s	LLM, the only option
Open-ended Q&A	n/a	30–60 s	LLM, the only option

These are single-machine timings from one session, so read them as orders of magnitude rather than precise ratios.

	Script	LLM
Nature	Deterministic	Probabilistic
Accuracy	Same every run, if coded right	My rough estimate: 80–95% depending on context
Latency	Milliseconds to seconds	20 to 600+ seconds on this hardware
Hallucination	Not possible	Possible
Maintenance	Edit the script	Edit the skill docs

The accuracy row matters most for me. When a task touches real money or real infrastructure, a wrong answer costs more than a slow one.

The architecture that came out of it

ORCHESTRATOR (cloud LLM)
  • understands intent
  • picks the right script
  • composes the final reply
  • troubleshoots failures, writes new code when needed
      │
      ├─► DATA tier     scripts that query business systems and export reports
      ├─► INFRA tier    proxmox_status.py, system_health.sh, ad-hoc SSH
      ├─► DOCS tier     scripts that wrap the Notion API
      └─► UTILITY tier  spreadsheet/PDF generators, file operations, the RAG pipeline

The decision tree is short: is it data retrieval, an infrastructure check, or a document operation? Then run a script. Only if it genuinely requires reasoning does it go to an LLM, and even then the LLM plans and explains while a human (or a script) executes.

Rules for adding a script

Read-only by default. Writes (saving a file, exporting a sheet) need an explicit --write or --excel flag.
Every script has --json for machine-readable output.
Credentials from the environment, never hardcoded.
One script per domain, one place to fix bugs.
Test on creation: run --help and one real call before relying on it.

Why not just use the LLM for everything?

On this hardware, a “Hi” takes 3 s, a 368-token RAG prompt takes 40 s, and an 8000-token prompt takes minutes. A script that logs in, queries, filters and writes a spreadsheet takes 2 s, and produces the same answer every time. Scripts have no hardware dependency. The LLM does.

That doesn’t make the LLM less useful — it makes it more precise about where it belongs. A script can’t tell you why a request timed out, or write the script you don’t have yet. That’s the LLM’s job.

Principles I kept

LLM for reasoning, scripts for execution. Never the other way round.
Deterministic beats probabilistic whenever the output is structured data.
Speed is a feature. A 1-second script beats a 60-second LLM that gives the same answer.
Accuracy is not negotiable when errors cost real money.
One source of truth per domain.

A companion experiment, Does Memory Plus Skills Close the Gap for a Small LLM?, looks at the other half of the same idea: giving a small model the knowledge it lacks.