Details
Observed result
Local repro typically ends with:
- a recovered-token trace showing:
sample_recovered_tokens_return -> recovered_token_ids[0] = 151936prepare_next_token_ids_padded -> next_token_ids[0] = -1set_inputs_first_pass -> input_ids_after[0] = -1embed_input_ids_out_of_range -> input_ids[0] = -1CUDA error: device-side assert triggered- a fatal engine-side failure
gRPC repro typically ends with:
- the triggering gRPC requests failing with
INTERNAL: EngineCore encountered an issue. See stack trace (above) for the root cause. - server logs showing the worker dies with
CUDA error: device-side assert triggered - a later public probe request also failing after the worker is dead
This demonstrates that the issue is reachable through the public gRPC request surface, not only through a local reproducer.
Log snippets
Local recovered-chain trace
sample_recovered_tokens_return:
recovered_token_ids = [151936, ...]
vocab_size = 151936
prepare_next_token_ids_padded:
sampled_token_ids_head = [[151936, -1, -1, ...], ...]
next_token_ids = [-1, ...]
set_inputs_first_pass:
input_ids_after = [-1, 0, 0, 0, ...]
embed_input_ids_out_of_range:
input_ids = [-1, 0, 0, 0, ...]
gRPC server log
torch.AcceleratorError: CUDA error: device-side assert triggered
...
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
...
Error in Generate for request post_crash_probe
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
Root cause
This is a speculative-decoding state-handling bug, not an invalid frontend token-id input bug.
The root cause is that a recovered speculative token can become equal to vocab_size, then be selected as the live next token for a request, then be converted to -1, and that converted -1 is still written back into live drafter input ids and later consumed by the drafting / embedding / attention path.
For the Qwen3 checkpoint used here:
151936 == vocab_size
This value should be described as the model vocab_size boundary value, not as a legal token id.
Attachments
The attached bundle for this report should contain:
repro_g4_recovered_minus1_local.pyrepro_g4_recovered_minus1_grpc.py
These two standalone scripts are sufficient to reproduce the issue and its public gRPC reachability.
Fix
A fix for this vulnerability has been merged in: https://github.com/vllm-project/vllm/pull/44744