Field note · DGX Spark
pgrep -f matched my own shell, and nohup did not survive the timeout
Two process-management traps that both look like a crashed inference server. pgrep -f matches the command line of the process doing the grepping, so a stop script can kill its own shell and a wait loop can "confirm" a server stopped when it did not. And nohup guards against SIGHUP, not the SIGTERM a timing-out parent sends to its process group — so a 90-second model load inside a 2-minute command budget is killed at exactly two minutes and looks identical to a load failure. Have the child write its own PID, and use setsid.
- Hardware
- NVIDIA DGX Spark · GB10 (SM121) · aarch64 · 128 GB unified
- Software
- bash · llama.cpp llama-server (~90 s model load, 93.6 GB across 3 shards)
1. pgrep -f matches the process doing the grepping
pgrep -f "llama-server.*--port 8001"The pattern is itself part of the command line of the shell evaluating it, so -f (match against the full command line) finds it. I hit this twice in one afternoon, in both directions:
- A stop script killed a PID that was its own shell.
- A wait loop "confirmed" the server had stopped, because the only surviving match was the loop itself — which then exited, leaving an 87 GiB process running that everything downstream believed was gone.
The usual mitigations are all partial. pgrep -f "[l]lama-server" works until someone reformats the string. pgrep -x drops -f and stops matching on arguments, which is exactly what you needed to distinguish two servers. Excluding $$ misses the subshell.
The fix that actually holds is to stop pattern-matching at all. Have the child write its own PID:
sh -c 'echo $$ > "$PIDFILE"; exec llama-server ...'exec replaces the shell, so the PID in the file is the server's. There is nothing to match, and nothing to get wrong when the argument list changes.
This exact failure was already written down in the repository it happened in, and I walked into it anyway — which is its own small lesson about where documentation stops helping.
2. nohup does not survive a timing-out parent
nohup ignores SIGHUP. A supervisor, CI step, or agent tool that enforces a command budget sends SIGTERM — usually to the whole process group — and nohup does nothing about that.
The specific shape: a model load takes about 90 seconds. The command budget is two minutes. Start the server, poll for readiness, and if anything is a little slow, the parent times out at 120 s and terminates the group. The server dies mid-load with no error of its own, which looks exactly like a crashed load.
setsid ./start.sh # new session, new process groupsetsid detaches into a new session so the parent's group-directed SIGTERM never reaches it. Combine with the PID file above and the server is genuinely independent of whatever launched it.
Why both matter more on a large model
Neither trap is specific to inference. They are worse here because the process is slow to start (~90 s) and slow to die: tearing down CUDA buffers and unmapping 87 GiB takes more than five seconds and completes well inside thirty. That window is long enough for a fixed sleep 2 after SIGKILL to produce a false driver-hang diagnosis on a process that is merely busy dying, and long enough for a restart to launch a second copy alongside the first — which on this host got an unrelated editor OOM-killed.
Poll for exit rather than sleeping, poll after SIGKILL too, and block on MemAvailable recovering rather than on the PID disappearing. The longer version, including why the kernel picks the inference server first, is in the kernel kills your inference server first.
