Field note · DGX Spark
DefaultOOMScoreAdjust=200 in the systemd user manager kills terminal-launched servers first
A production llama-server process was killed by the kernel with nothing in its own log: no assert, no stack trace, just silence, noticed only because /metrics stopped responding. The killed process carried oom_score_adj:200. The cause was not the coding-agent session that launched it — it was a systemd user-manager default, DefaultOOMScoreAdjust=200, which scores every terminal-launched process to die before system services do, even ones many times smaller. That score cannot be lowered after launch by an unprivileged process, so the fix is to run the server as a systemd system service instead, which defaults to OOMScoreAdjust=0.
- Hardware
- NVIDIA DGX Spark · GB10 (SM121) · aarch64 · 128 GB unified
- Software
- llama-server launched from an interactive shell under the systemd user manager (DefaultOOMScoreAdjust=200)
Out of memory: Killed process 1966928 (llama-server) total-vm:164222812kB anon-rss:31651260kB oom_score_adj:200Symptom
A production llama-server process disappeared mid-run with nothing in its own log — no assert, no stack trace, not even a final line. RSS is the wrong instrument for memory on a DGX Spark covers how that kill was found and what it meant for memory headroom. This note is about one detail in the kernel's own record of it, quoted above: oom_score_adj:200. That number is not small on this host, and it is the reason this particular process — not something else running at the same time — was the one the kernel chose to kill.
The wrong first guess
The obvious guess was that llama-server had inherited 200 from the coding-agent session that launched it — some score passed down the process tree from an agentic wrapper. That guess was wrong. Relaunching the identical server from a plain login shell, no agent anywhere in the tree, produced the same 200. The source was systemd itself:
$ systemctl --user show -p DefaultOOMScoreAdjust
DefaultOOMScoreAdjust=200DefaultOOMScoreAdjust=200 is the default set by the systemd user manager on this host. It applies to every process launched from a user session — an interactive terminal, an SSH login, a coding agent, anything — regardless of what that process actually is or how much memory it needs.
The resulting hierarchy is inverted
| process class | oom_score_adj | killed... |
|---|---|---|
system services (incl. dgx-dashboard) | 0 | last |
gnome-shell, user systemd | 100 | |
| anything launched from a terminal | 200 | first |
On a box whose entire purpose is running an inference server, that ordering is backwards. The kernel reaches for the largest, most important process on the machine — 87 GiB of it, in this case — before it would ever touch a system service a fraction of that size. In the incident above, a small system service (dashboard-servi) made the allocation that tipped the box over its memory limit; it did not cause the pressure, it just triggered the kernel's OOM path, and llama-server lost purely on score, not on which process was actually responsible for the shortage.
It cannot be fixed after the process is already running
$ echo 0 > /proc/<pid>/oom_score_adj
DENIED — still 200An unprivileged process may raise its own oom_score_adj. It may not lower it. Once a server is running with a 200 inherited from the user manager, nothing short of relaunching it under a different path changes that.
The fix
Run the inference server as a systemd system service, not from an interactive shell. System services default to OOMScoreAdjust=0, which puts the server on equal footing with the rest of the system instead of volunteering it first. Short of that, know that on this platform any terminal-launched long-lived process carries a 200 and budget memory headroom accordingly — and if a long-running server vanishes with nothing in its own log, check cat /proc/<pid>/oom_score_adj and journalctl -k | grep -i oom-kill before looking for anything else.
None of this is specific to this model, this inference engine, or even to DGX Spark. Any host that runs systemd with a per-user manager, and launches its most important long-lived process from a terminal instead of as a system unit, has the same inversion waiting.