Two ways a DGX Spark inference server dies after hours of clean operation

torch.AcceleratorError: CUDA error: an illegal instruction was encountered

Variant 1: the trap (Mistral Small 4 119B NVFP4, ~10 hours)

This is what's left after the FP8 arch-guard fix. That patch moved the same top-level CUDA fault from a ~12-minute crash to something that survives roughly ten hours of clean operation before failing a different way.

The server came up around 07:25 UTC and served traffic without incident all day. The last successful 200 OK landed at ~17:16 UTC; the crash hit thirty minutes later, at 17:46, mid-request:

num_computed_tokens=[40149]
num_output_tokens=[3]
has_structured_output_requests=true
GPU KV cache usage: 9.2%

Forty thousand computed tokens and three output tokens — nowhere near a KV-cache ceiling at 9.2% utilization. The crash trace pointed at rms_norm again, reached through the same MLA path as the original FP8 trap (mla.py:151 → kv_a_layernorm(kv_c)), and for the same reason that trace was misleading the first time: CUDA reports these errors asynchronously, so the reported site is wherever the host next happened to check, not necessarily where the bad kernel actually ran.

None of the working theories from that day are confirmed — another enable_sm120_only-style guard buried somewhere in a compiled dependency (flashinfer, a CUTLASS submodule), an SM121 kernel bug specific to large tensor dimensions (40k+ tokens), unified-memory pressure after hours of uptime, or the guidance/Lark grammar backend hitting an incompatible kernel path. CUDA_LAUNCH_BLOCKING=1 went into the start script to catch the next one with a real kernel name attached.

Variant 2: the silent hang (Qwen3.6 int4, ~7 hours)

A different model, a different failure shape, found about four weeks later. Intel/Qwen3.6-35B-A3B-int4-AutoRound, served with --moe-backend cutlass and MTP=2, started at ~14:30 UTC and ran clean until ~21:44 — about 7 h 14 min of uptime, earlier than variant 1's crash and by a completely different mechanism.

Nothing crashes. /health returns 200. /v1/models returns 200. A new request is accepted and enters the scheduler — Running: 1 — and then just sits there:

Engine 000: Avg prompt throughput: 2.2 tokens/s, Avg generation throughput: 2.8 tokens/s, Running: 1 reqs (21:44:28)
Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs (21:44:38)

Ten seconds apart. Generation throughput drops to 0.0 tok/s and stays there; GPU utilization falls to 3%; no traceback, no illegal instruction, no container exit. New requests queue up behind the stuck one and never start generating. The eighteen minutes before the visible hang (21:26–21:44) already showed only intermittent activity, which reads more like gradual degradation than an instant flip.

There was a signal ahead of the hang: MTP acceptance rate had fallen to 53–62% in the final hour, against 74–83% on a fresh boot. Whether that's cause, correlate, or coincidence isn't established — only that it's a number worth watching next time.

The fix, such as it was, was operational rather than diagnostic: kill the container and restart on a newer vLLM build (c798593f0). Fresh-boot acceptance came back to 74–83%, and thirty-plus minutes under bench load showed no recurrence. That resolves the symptom. It doesn't explain it.

One family, or two bugs?

I don't know, and I'm not going to manufacture a conclusion this note hasn't earned. One variant traps with an illegal instruction. The other deadlocks silently. Both surface only after extended operation on the same SM121 hardware — but "extended" means two different things here: ~10 hours for the NVFP4 trap, ~7 h 14 min for the int4 deadlock. That gap may mean the threshold scales with cumulative generated tokens, or with the specific MoE kernel mix each model exercises, rather than with wall-clock uptime at all. It might equally mean these are two unrelated bugs that both happen to need hours of sustained load to surface, and SM121 simply has more than one of those waiting.

The next diagnostic step is the same one already used on the trap side: CUDA_LAUNCH_BLOCKING=1, added to the start script after that crash. It's still off in the Qwen path (qwen36-start-local.sh) — turning it on there is what would finally attach a real kernel name to the next hang.