The stack I actually run.
Not a skills list — this is the hardware running under my desk right now, the two inference servers serving models off it, and the web stack that ships everything else I build.
Hardware
- System
- NVIDIA DGX Spark
- GPU
- GB10 (SM121, CUDA compute 12.1a)
- Memory
- 128 GB unified (shared CPU/GPU)
- Architecture
- aarch64 (ARM)
- CUDA
- 13.0
- Driver
- 580.142
SM121 is the DGX Spark variant of Blackwell. It differs from SM120 — the RTX 5090 — in subtle ways that make some pre-compiled kernels crash outright. See every FP8 GEMM trapping on SM121 for the specific failure this caused and the two-line fix.
The 128 GB is genuinely unified: the Linux page cache competes with the GPU for it directly, so a large Docker build or file copy can starve a model mid-load with a CUDA OOM even while free -h still shows memory free.
Serving
Two engines, two ports, one piece of hardware.
vLLM
Built from a spark-fixes branch that tracks vLLM main, carrying one patch: the FP8 enable_sm120_family fix. Two lines in csrc/quantization/w8a8/cutlass/c3x/ widen an arch guard that upstream pins to an exact SM120 match — and traps on anything else — to accept the whole SM120 family instead. Compiled with torch_cuda_arch_list="12.1a" against CUDA 13.0.1; Mistral and Gemma 4 serve from the resulting vllm-mistral:cu130 image, on port 8000.
llama.cpp
A native build, not a container: cmake with -DGGML_CUDA=ON against the same aarch64 / CUDA 13.0 toolchain, arch pinned to 121a-real— GGML's spelling of the same SM121 target vLLM calls 12.1a. Produces a single llama-server binary serving GGUF models with an OpenAI-compatible API on port 8001. Every model runs with --kv-unified: unified CPU+GPU memory for the KV cache, which matters here because there is no separate GPU memory to unify away from.
No SM121-specific source patches have been needed for llama.cpp itself — unlike vLLM's FP8 guard — though this exact aarch64 / CUDA 13.0 toolchain has already produced one build failure on this box: a KV-cache batching call with the wrong CUDA signature.
Models
What is actually being served right now, on both engines.
vLLM models — port 8000
Daily driver: Qwen3.6 35B-A3B, served int4 AutoRound-quantized. current daily driver
| Model | Size | Notes |
|---|---|---|
| Mistral Small 4 119B NVFP4 | ~66 GB | MoE, 6.5B active params · --moe-backend cutlass (why this backend) |
| Gemma 4 31B IT NVFP4 | ~31 GB | Dense · reasoning via --reasoning-parser gemma4 · tool calling via --tool-call-parser gemma4 (needs PR #38847 or a local fix) |
Mistral's own numbers on this hardware: ~26 tok/s single-request, ~60 tok/s total at 4 concurrent requests, 81%+ prefix-cache hit rate, 262,144-token context.
llama.cpp models — port 8001
All seven run at a 262,144-token context with multimodal (vision) support via --mmproj.
| Model | Type | Size | Quant | Slots | Notes |
|---|---|---|---|---|---|
| Gemma 4 26B-A4B-it | MoE — 26B total, ~4B active | ~16 GB | UD-Q4_K_XL | 4 | Lightweight, multimodal |
| Gemma 4 31B-it | Dense — 31B all active | ~18 GB | UD-Q4_K_XL | 2 | Deeper reasoning than the 26B MoE |
| Mistral Small 4 119B | MoE — 119B total | ~73 GB | UD-Q4_K_XL | 1 | Also supports an MXFP4_MOE variant |
| Qwen 3.5 122B-A10B | MoE — 122B total, ~10B active | ~77 GB | UD-Q4_K_XL | 4 | Also MXFP4_MOE; control-vector support |
| Qwen 3.6 35B-A3B | MoE — 35B total, ~3B active | ~22 GB | UD-Q4_K_XL | 4 | Thinking on by default |
| Qwen 3.8 27B Uncensored | Dense hybrid GDN+attention, vision | ~29 GB | Q8_0 | 4 | MTP N=3 plus vision |
| Step-3.7-Flash | StepFun fork tree | ~89 GB | — | 1 | Not mainline llama.cpp |
Web stack
The other half of what I build — what actually ships to a browser.
- TypeScript
- React
- Next.js
- Tailwind
- Node.js
- AWS (Lambda, SES, Amplify)
- Playwright
- Jest
Related field notes
The stack above breaks in specific, documented ways. Three of the most recent:
llama.cpp
resolvedDownloading one new file from a Hugging Face repo invalidates the path to all the others
Fetching a newly published mmproj moved refs/main to a fresh snapshot directory containing only that one file, while 93.6 GB of weights stayed in the previous snapshot. Any launcher resolving through refs/main — which is the recommended practice, since snapshot hashes change on every re-pull — then fails its own existence check on weights that are plainly on disk. The fix downloads nothing: a second hf download with the original include pattern relinks the existing blobs into the current snapshot.
August 28, 2026
llama.cpp
resolvedDropping --kv-unified silently turns a 262144-token server into a 65536-token one
llama-server divides the context size by the slot count unless --kv-unified is set: n_ctx_seq = n_ctx / n_seq_max. There is no error and no warning. The same command line that asks for --ctx-size 262144 --parallel 4 serves 65536 per sequence, and the only evidence is one line in the boot log. Long prompts then fail or shift in ways that look like a model problem rather than a configuration one.
August 28, 2026
llama.cpp
workaroundGGML_ASSERT(layer.nextn.hc_head_norm) — two MTP pull requests disagree about one tensor name
A published Qwen3.8-Flash-Next MTP head loads against PR #27739 and aborts against PR #27836 with "MTP block missing nextn.hc_head_norm". The head is not damaged and no tensor is missing: the same three hyper-connection tensors are exported as top-level output_hc_norm/down/up by one PR and expected as blk.N.nextn.hc_head_norm/down/up by the other, because a standalone head file has no trunk to collide with and a grafted head does. The published merge script filters to blk.* and silently drops all three. Rename them instead of dropping them.
August 28, 2026