Running a Local LLM on CPU: llama.cpp, Qwen3, and Open WebUI

Every request to a hosted model is a request that leaves the network. For a lab that touches client infrastructure, that is not always acceptable — so I wanted a local model good enough for tool-calling and document work, running on hardware I already had.

No GPU. One VM, CPU inference, OpenAI-compatible API, chat UI in front.

Measured result: 8.7 tokens/s generation, ~18 tokens/s prefill on 16 vCPU. Slow enough to notice, fast enough to use for anything asynchronous.

The VM

qm clone 9000 115 --name llm-local --full
qm set 115 --memory 32768 --balloon 16384 --cores 16 --sockets 1 \
  --cpu host --hotplug cpu,memory,network,disk --numa 1 \
  --ciuser faiez --ipconfig0 ip=10.10.10.81/24,gw=10.10.10.1 --nameserver 10.10.10.1 \
  --sshkeys <pubkey>
qm resize 115 scsi0 300G
qm start 115

Mistake #1: cpu=host is mandatory. The default kvm64 model does not expose AVX2, and llama.cpp silently falls back to slower kernels — roughly half the throughput — with no warning. If your CPU inference is mysteriously slow on a VM, check grep avx2 /proc/cpuinfo inside the guest before tuning anything else.

Mistake #2: memory hotplug requires NUMA. --hotplug memory without --numa 1 fails at start with NUMA needs to be enabled.

Engine and model

# llama.cpp prebuilt (ubuntu-x64 tarball) — needs libgomp1
sudo apt install -y libgomp1

Model: Qwen3-14B-Instruct, Q4_K_M quantization, ~9 GB.

Why 14B and not 8B: tool-calling reliability is the requirement, and it degrades faster than general chat quality as models shrink. Why Q4_K_M: a 14B fp16 model is ~28 GB; 4-bit quantization brings it to 9 GB with negligible quality loss for this workload. Think of GGUF quantization as MP3 for weights.

Serving config:

llama-server \
  --model /home/faiez/models/Qwen3-14B-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  -t 16 --mlock --jinja \
  --chat-template-kwargs '{"enable_thinking":false}' \
  -c 16384

Mistake #3: thinking mode breaks tool calls. Qwen3 emits its reasoning into reasoning_content first. With a small max_tokens, the response or the tool call never gets emitted — the API returns an empty assistant message and you blame the client. Disabling thinking via --chat-template-kwargs makes tool calls fast and reliable:

{"role": "assistant", "tool_calls": [{"function": {"name": "weather_check", "arguments": "{\"city\":\"KL\"}"}, "id": "call_abc"}]}

--mlock keeps the 9 GB of weights resident. Without it, page reclaim on a 32 GB VM produces latency spikes that look like model instability.

Systemd unit so it survives reboots, then:

systemctl status llama-server
journalctl -u llama-server -f

Measured performance

For context on what 8.7 tok/s means: a 500-token answer takes about a minute. That is fine for background jobs, summarisation, and agent loops that are doing other work. It is not fine for interactive chat, and pretending otherwise leads to a bad design.

The chat UI

Open WebUI in Docker on the agent host, pointed at the llama.cpp endpoint:

docker run -d --name open-webui --restart always \
  -p 3001:8080 -v open-webui-data:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Then in the UI: add a connection pointing at http://<llm-host>:8080/v1, any non-empty API key.

Browser → Open WebUI (3001) → llama.cpp (8080) → Qwen3

Two operational notes: port 3001, not 3000 — 3000 was already taken by a WhatsApp bridge, and the container will start fine while binding nothing you expect. And register the admin account immediately, then disable public sign-up, because an open Open WebUI instance is an open proxy to your model.

An 8B Qwen3 Q4_K_M is a reasonable alternative if RAM is tight: better latency, worse tool-call reliability. On a 32 GB VM, run the 14B.

What this is actually good for

It replaced hosted API calls for anything sensitive — document handling, internal notes, anything containing client names — while keeping the hosted model for everyday work where speed matters.

The honest summary: a CPU-only 14B is a privacy tool, not a performance tool. 8.7 tok/s is not a limitation to engineer around; it is the constraint that decides which workloads get routed locally.

The obvious next step is a GPU. The VM already has hotplug and NUMA configured, and IOMMU/VT-d is active on the host — so the upgrade path is a passthrough device, not a rebuild.