Every FP8 GEMM traps on DGX Spark (SM121)

Symptom

Serving mistralai/Mistral-Small-4-119B-2603-NVFP4, the model loaded cleanly — 66.22 GiB in 488 seconds — and handled several requests successfully before anything went wrong. Two flags were new since the last known-good run, --async-scheduling and --enable-chunked-prefill, and they looked like the obvious suspects. They turned out to be unrelated.

The crash landed about twelve minutes into generation, throughput at 15–16 tok/s, on a request that had already produced 29,583 computed tokens and 1,719 output tokens. GPU KV cache usage was 7.9% — nowhere near a resource ceiling. The reported trace pointed at rms_norm in layernorm.py:61 (out = torch.empty_like(x)), reached through the MLA attention path: deepseek_v2.py:1005 → mla.py:151 → kv_a_layernorm(kv_c) → rms_norm.

Treat that trace with suspicion. rms_norm is a LayerNorm op — it has no obvious connection to FP8 quantization at all, and that mismatch is itself the tell. CUDA kernel launches are asynchronous: a fault raised inside one kernel is often only detected, and attributed, at the next point the host happens to check for errors. The line vLLM reports is wherever that check landed, not necessarily where the illegal instruction actually executed.

Root cause

The actual fault was nowhere near rms_norm. vLLM's FP8 CUTLASS kernels are gated by a compile-time guard, enable_sm120_only, that reduces to something like this:

#if __CUDA_ARCH__ == 1200
  // real kernel body
#else
  asm("trap;");   // deliberate illegal instruction
#endif

DGX Spark's GB10 identifies as SM121 — __CUDA_ARCH__ reports 1210, not 1200 — so the guard's else branch is what actually runs for any FP8 GEMM compiled against it on this hardware. Despite what a twelve-minute delay might suggest, this isn't a race condition, a memory fault, or an edge case that only shows up in long-running generation. It's a guard written for one specific architecture number, checked with equality instead of a range, on hardware whose numeric identity is one past it. The build even targeted both architectures (torch_cuda_arch_list="12.0 12.1"), which made this more confusing, not less — broad arch coverage at the build-flag level doesn't help when a header two layers down hardcodes == 1200. NVIDIA discusses the SM120/SM121 split in its own developer forum.

Fix

Two files, one guard swapped in each — an equality check replaced with a range check:

csrc/quantization/w8a8/cutlass/c3x/scaled_mm.cuh
csrc/quantization/w8a8/cutlass/c3x/scaled_mm_sm120_fp8_dispatch.cuh

// enable_sm120_family(), in place of enable_sm120_only():
__CUDA_ARCH__ >= 1200 && __CUDA_ARCH__ < 1300

enable_sm120_family accepts the whole SM120.x line instead of exactly one member of it. Committed as 228f0086a on my mistral-fixes-v2 branch. Every FP8 GEMM that used to trap on this hardware now runs.

What it does not fix

This patch buys hours, not immunity. After it landed, the same server ran for roughly ten hours under load instead of twelve minutes before crashing again — real progress, and a completely different failure once it happened. That one traps somewhere else, on a much longer timescale, and whether it shares a root cause with this one is still an open question. It gets its own note: Two ways a DGX Spark inference server dies after hours of clean operation.