GPU & performance

ArcFlare finds every llama.cpp build on the machine and probes each one for devices:

  OK  vulkan:vulkan        ~/llamacpp/vulkan  <- active
      Vulkan0  AMD Radeon(TM) 8060S Graphics (47.8 GB, 45.4 GB free)
  --  rocm:rocm            ~/llamacpp/rocm
      build loads but enumerates no device (driver or runtime)
      kernels present: gfx1010 ... gfx1151 ... gfx1250
      the build is fine - the driver is not exposing the GPU to HIP.

That last line is the useful part. The ROCm build above does contain kernels for this GPU, so reinstalling it would not help, and the driver is what needs updating. Select a backend with arcflare backend vulkan (or cuda, rocm), and re-probe with --probe.

llama-server's stock defaults assume RAM to spare: an 8 GB host prompt cache and up to four resident models. On a machine whose RAM is mostly carved out for VRAM, that is the biggest avoidable cost.

The physical batch (-ub) trades VRAM for prefill speed, and the trade goes both ways. Measured on a 35B-A3B at the full 262,144 context:

Agent turns skew short, because a cached session only re-prefills what changed. So the default is 2048. arcflare batch 512 reverses it if your work is long single prompts. The choices are 512, 1024, 2048 and 4096.

What fits is arithmetic over the GGUF header and your free VRAM. It covers every model, instantly, with nothing loaded:

  What fits  45.4 GB free on device

  + qwen3.6-35b-a3b:ud-q6_k_xl  30.4 GB  20 KiB/tok  256K  full 256K context - 5.0 GB of cache, 8.7 GB left over
  + qwen3.8-27b:ud-q5_k_m       18.4 GB  64 KiB/tok  256K  full 256K context - 16.0 GB of cache, 9.6 GB left over

What runs best can't be computed. --run loads each model at the context it would really use and times it:

  qwen3.6-35b-a3b:ud-q5_k_xl  23.6 s  610 tok/s  57.5 tok/s  5/5 probes  256K ctx
  qwen3.8-27b:ud-q5_k_m       31.8 s  176 tok/s  19.4 tok/s  5/5 probes  256K ctx

  > use         qwen3.6-35b-a3b:ud-q5_k_xl
    fastest of the 4 that answered every probe

A smaller model is not a faster model. The dense 27B above is the smallest file and generates at a third of the speed of a MoE twice its size. File size predicts whether a model fits. It says nothing about how fast it runs.

• Generation speed is reproducible run to run. Trust it.
• Prefill moves by ~10% between runs. Don't rank quants on a small gap.
• Load time mostly measures your disk cache.

Five probes then check that a model still works as a tool-caller: exact output format, clean JSON, arithmetic, a function that actually runs in a sandbox, and recall from 4k tokens back. The recommendation is the fastest model that passes all five. Speed never makes up for a failed probe.

Results are kept in ~/.arcflare/fit.json and saved after each model, so an interrupted run keeps what it measured.