[ docs ] // Running models

Models & context

ArcFlare does not keep its own model store. It reads the GGUFs you already have, works out what each one costs to run, and offers the largest context that will really fit.

Where models are found#

Searched in this order, and every root is read:

Location	What it is
$ARCFLARE_MODELS	Your own folder, if you set one
$LLAMA_CACHE	llama.cpp's download cache
~/.arcflare/models	ArcFlare's default
~/llamacpp/models	A common hand-made layout
platform cache	llama.cpp's per-OS default cache directory

ArcFlare understands the Hugging Face cache layout, groups multi-part shards into one model, and picks up sidecar files next to a model: mmproj-*.gguf for vision and mtp-*.gguf drafts for speculative decoding.

arcflare ls

  qwen3.6-35b-a3b:ud-q6_k_xl  Q6_K · 30.4 GB · 256K ctx · MoE 8/256
  qwen3.6-35b-a3b:ud-q5_k_xl  Q5_K_M · 25.3 GB · 256K ctx · MoE 8/256   57.5 tok/s
  qwen3.8-27b:ud-q5_k_m       Q5_K_M · 18.4 GB · 256K ctx

A measured speed from arcflare fit --run shows up in this list, because a measurement beats anything a file header can tell you.

Drafts are matched across repos, vision projectors are not

The Hugging Face cache often splits one model over two repos, for example a draft from ggml-org and the weights from unsloth. So a speculative-decoding draft is matched to any model of the same family. An mmprojis repo-local, though, and pairing one with another model's weights would be a real mistake, so those are never matched across folders.

Pulling a model#

arcflare pull unsloth/Qwen3.6-35B-A3B-GGUF:Q5_K_XL

The format is user/repo[:QUANT]. The download is done by llama.cpp's own downloader and lands in $LLAMA_CACHE (or the first model root), where arcflare ls finds it.

Names are matched loosely. arcflare run qwen3.6picks the closest model. A name that matches nothing is an error, never "whichever model sorts first".

How the context is sized#

ArcFlare reads each model's GGUF header, works out what a given context costs in KV cache, and offers the largest size that fits your free VRAM. The model's trained maximum comes first.

It accounts for hybrid attention. On Qwen3.5/3.6/3.8-class models only every Nth layer keeps a growing KV cache, and the rest are linear-attention layers with fixed-size state. Assuming every layer attends overestimates the cache by 4×, and makes a maximum context look impossible when it isn't.

Model	Layers	Attending	KV / token (f16)	256K context
Qwen3.8-27B	65	16	64 KiB	17.2 GB
Qwen3.6-35B-A3B	41	10	20 KiB	5.4 GB

MLA models (DeepSeek, GLM-lite) are handled too. There the cache is one compressed latent per layer, not separate K and V.

Verify, don't predict#

Reported free memory is a budget, not a promise. llama.cpp has printed the same "free" figure while a 30 GB model was resident and right after that process exited. So ArcFlare cross-checks against what the OS reports as in use, then loads the model, and backs off if the device still refuses:

  loading Qwen3.6-35B-A3B @ 256K ctx...
  . did not fit - retrying at 128K
  + loaded at 128K context  (reduced from 256K - device could not fit it)

The physical batch is given up first: it shrinks to 512 and retries at the same context.
Only then is the context halved. A few percent of prefill speed is worth far less than the window it buys.
The result is the largest context that genuinely loads, not a confident number and an out-of-memory error on the first request.

Sampling#

Qwen3-class GGUFs publish the sampling they were tuned for under general.sampling.*, and their thinking and non-thinking presets differ (top_p 0.95 against 0.8). ArcFlare uses what the file declares, and only falls back to its own defaults when a model declares nothing.