Homelab AI infrastructure
The local infrastructure behind my AI projects: a place to run models, reproduce failures, and turn the fixes into useful field notes.
- Who it is for
- My own AI projects, and other builders facing the same local-inference failures.
- My role
- Owner and operator — model serving, debugging, evaluation, and documentation.
- Where it stands
- In daily use for local AI work. Compatibility and reliability remain ongoing work.
NVIDIA DGX Spark · vLLM · llama.cpp · Linux · CUDA · Python
The problem#
I wanted to understand the systems my AI projects depend on: what fits in memory, which workloads run reliably, and what fails after the health check passes. Running the models myself makes those questions part of the work.
What I built#
The lab centers on an NVIDIA DGX Spark, with vLLM and llama.cpp serving local language models. It supports The Agentic Daily, the writing pipeline, and the experiments behind this site's blog and field notes.
The current hardware and software choices live on the setup page. The most useful public output is the record of failures, measurements, and fixes.
Decisions that mattered#
- Run the stack I am trying to understand. Local serving makes memory, quantization, scheduling, and failure behavior things I can investigate directly.
- Make repeated experiments affordable. Owning the hardware removes per-token API billing from these local runs. That makes it easier to run enough replicates to test an evaluation.
- Publish the specific failure. A reproducible symptom, an explanation, and a bounded fix are more useful to another builder than a list of installed tools.
What I learned#
New hardware does not mean the software is ready for it. One FP8 failure came from kernels guarded for SM120 executing a trap on the Spark's SM121 GPU. The fix widened an architecture check.
Failures can also come from outside the model. In the inference-server postmortem, an out-of-memory kill exposed how Linux was choosing which process to sacrifice. Looking only at the model's own log would not have explained it.
The next question#
Can I recognize a failing workload before a reader or another system depends on its answer? A process can be alive and still do no useful work. The lab is where I keep learning what needs to be observed beyond a successful startup or a green health check.