llamaserver.nix

A NixOS module for serving GGUF models with llama.cpp llama-server in router mode. You declare models and per-model settings as structured options; the module generates everything else — the systemd service, the --models-preset INI, the service user. The pinned CUDA build of llama-server comes from this flake.

Models are identified by their Hugging Face location (repo + file, full URL, or a local path). The module does not download them: llama-server fetches hf-repo/hf-file models on demand (first time a client requests that model) into the standard HF hub cache under the instance's state dir.

How it works

• One option block per server: services.llama-server.<name>.
• Every instance is one llama-server in router mode: --host … --port … --models-preset /etc/llama-server-<name>/models.ini --models-max N --metrics [--sleep-idle-seconds S].
• The module generates /etc/llama-server-<name>/models.ini via environment.etc. Each model is a [name] section holding its source (hf-repo/hf-file or model = /path) and its per-model settings (ctx-size = 102400, ngl = 99, …). Instance-wide settings land in the [default] section and apply to every model.
• --models-max 1 (default) = LRU: with one model loaded at a time, a single card can serve several large models. Idle unloading via sleepIdleSeconds.
• First request for a model triggers its download into <stateDir>/hf (HF hub layout, HF_HUB_CACHE is pinned there). Reboots are instant — models are already cached; the cache survives config changes.
• Gated repos: set hfTokenFile to a file containing the HF token (readable by the service user). The module wraps the binary so HF_TOKEN is exported for the server process.

Wiring it into a NixOS host

Add this flake as an input and import its module:

# flake.nix inputs
inputs.llamaserver.url = "path:/path/to/llamaserver.nix";

# system modules
modules = [ llamaserver.nixosModules.default ];

Then declare your servers:

services.llama-server.llm = {
  enable = true;
  port = 8080;
  # modelsMax = 1;          # LRU swap (default)
  # metrics = true;        # /metrics (default)
  sleepIdleSeconds = 120;
  # hfTokenFile = /run/llama-hf-token;  # only for gated repos
  settings = {              # instance-wide, [default] section
    threads = 4;
    jinja = true;
  };
  models = [
    {
      name = "my-model";    # the alias clients request
      repo = "org/my-model-gguf";     # <org>/<repo>
      file = "my-model-Q4_K_M.gguf";
      settings = {
        ctx-size = 102400;
        ngl = 99;
      };
    }
    # Full-URL form also works:
    # { name = "x"; url = "https://huggingface.co/org/repo/resolve/main/f.gguf"; }
    # Local file (no download):
    # { name = "y"; path = "/var/lib/llama-cpp/models/y.gguf"; }
    # With an MTP draft shipped as a separate HF repo:
    # { name = "z"; repo = "org/z"; file = "z.gguf";
    #   draftUrl = "https://huggingface.co/org/z-mtp/resolve/main/mtp-Q4_0.gguf"; }
    # With a local draft file:
    # { name = "w"; path = "/var/lib/llama-cpp/models/w.gguf";
    #   draftPath = "/var/lib/llama-cpp/models/mtp.gguf"; }
  ];
};

The server user (default llama, system user, home /var/lib/llama-cpp, groups video + render for /dev/nvidia*) is created automatically by the module.

HF token for gated repos

hfTokenFile just needs to point at a file readable by the service user that contains the Hugging Face token. How you materialize that file is up to your host (sops, a plain 0400 file, a secrets manager, …). The module only ever sees the path — it never reads the token itself.

Option reference

services.llama-server.<name>.* (all optional unless marked):

Option	Type	Default	Effect
enable	bool	false	no service/user/etc until true
package	package	this flake's llama-cpp-cuda (nixpkgs CPU build if module.nix is imported directly)	the llama-server binary
port	port	— (required)	--port
host	str	"0.0.0.0"	--host
modelsMax	int	1	--models-max (0 = unlimited; 1 = LRU)
metrics	bool	true	--metrics (Prometheus /metrics)
sleepIdleSeconds	int?	null	--sleep-idle-seconds
settings	attrset	{}	instance-wide flags → [default] section
user	str	"llama"	service user (created by the module)
stateDir	str	"/var/lib/llama-cpp"	user home; HF cache in <stateDir>/hf
hfTokenFile	path/str?	null	file with HF token → HF_TOKEN for gated repos
extraArgs	list of str	[]	raw extra flags (escape hatch)
models	list	[] (must be non-empty when enabled)	see below

models[i].*:

Option	Effect
name	preset alias clients request (<name> INI section)
url	full HF URL (…/resolve/<branch>/<file>.gguf, …/tree/…, or bare repo URL)
repo + file	HF <org>/<repo> + GGUF path (alternative to url); a bare repo URL also accepts a file
path	local GGUF file (no download)
draftUrl	HF repo of a separate draft model → spec-draft-hf (repo-level; llama.cpp picks the GGUF)
draftPath	local draft file → model-draft (e.g. an MTP head GGUF)
settings	per-model llama-server flags, keys are the flag names minus the leading -- (ctx-size, ngl, jinja, …); bool → true/false, everything else as-is, unquoted

Validation happens at eval time: exactly one of url/repo/path per model, unique model names, no spaces/newlines in names, file needs repo or url, models list non-empty. Anything the server doesn't recognize fails loudly at service start: option 'X' not recognized in preset 'name'.

What the module generates

• systemd.services.llama-server-<name> — Type=simple, runs as the instance's user (default llama), Restart=on-failure (5 s), LimitMEMLOCK=infinity (for load-mode = mlock), HF_HUB_CACHE=<stateDir>/hf, ExecStart = binary (or token wrapper) + flags.
• /etc/llama-server-<name>/models.ini — the generated preset (never hand-edit).
• users.users.<user> / users.groups.<user> (default llama) — created once per unique user across all enabled instances.

Building & checking

# module check: pure NixOS config eval with a stub llama-server; asserts the
# generated INI, ExecStart, and service user. Fast, no GPU, no CUDA build:
nix --extra-experimental-features "nix-command flakes" flake check

# the CUDA build (x86_64 with a GPU or for the store):
nix build .#packages.x86_64-linux.default -L

The check runs the module through nixos/lib/eval-config.nix with a stub binary, two instances sharing one user (dedup test) and one disabled instance (negative test).

Updating llama.cpp

Bump the tag in flake.nix (inputs.llama-cpp.url), nix flake update llama-cpp, rebuild, and smoke-test on the target host before migrating consumers. The preset key set comes from common/arg.cpp in the pinned source; if upstream renames flags, update settings keys accordingly (the server's own preset validation catches unknown keys at start).