IndexError in the multimodal processor when vLLM initializes Mistral Small 4

Symptom#

Serving mistralai/Mistral-Small-4-119B-2603-NVFP4, the server never gets as far as accepting a request. It crashes during initialization, while vLLM is building a dummy multimodal input to size its encoder budget:

File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 143, in __init__
    self.input_processor = InputProcessor(self.vllm_config, renderer)
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/input_processor.py", line 61, in __init__
    mm_budget = MultiModalBudget(vllm_config, mm_registry)
File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/encoder_budget.py", line 87, in __init__
    all_mm_max_toks_per_item = get_mm_max_toks_per_item(
File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/encoder_budget.py", line 32, in get_mm_max_toks_per_item
    mm_inputs = mm_registry.get_dummy_mm_inputs(
File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/registry.py", line 240, in get_dummy_mm_inputs
    mm_inputs = processor.apply(
File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/processing/processor.py", line 1685, in apply
    ) = self._cached_apply_hf_processor(inputs, timing_ctx)
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/pixtral.py", line 277, in _cached_apply_hf_processor
    prompt_ids, mm_info, _ = super()._cached_apply_hf_processor(inputs, timing_ctx)
File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/processing/processor.py", line 1496, in _cached_apply_hf_processor
    mm_kwargs, mm_prompt_updates = self._merge_mm_kwargs(
File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/processing/processor.py", line 1374, in _merge_mm_kwargs
    missing_kwargs_item = missing_kwargs[missing_next_idx]

Pixtral reaches that point through its own override of _cached_apply_hf_processor (pixtral.py:277) before landing in the shared _merge_mm_kwargs — which is why this is specific to the Pixtral architecture, not every multimodal model vLLM supports.

Root cause#

Introduced by commit 502c41a8f, "[Model] Use helper function to run MM processors with token inputs (where applicable)" (PR #38018). That change altered how multimodal processors handle token inputs, and the Pixtral architecture — the vision-language architecture behind Mistral Small 4 — crashes while computing dummy multimodal inputs during initialization, before any real request is processed. The commit immediately before it, 290809456, works fine with this model.

Fix#

Revert the regression:

git revert 502c41a8f

This is not a DGX Spark problem. The crash happens during model initialization, in shared multimodal-processing code, before any hardware-specific path is reached — it likely affects every Pixtral or Mistral Small 4 deployment running this vLLM revision, regardless of hardware. If you're hitting an IndexError at _merge_mm_kwargs during vLLM init on a Pixtral-family model, checking whether your build includes 502c41a8f is worth doing before looking anywhere else.