[Model] Ultravox: Support Llama 4 and Gemma 3 backends

[farzadab]

This is a simplified version of my older PR that was approved by @DarkLight1337 but ended up not working on some backends: https://github.com/vllm-project/vllm/pull/15728/files
This new PR allows Ultravox to support Gemma 3 and Llama 4 backends.

On the Ultravox side, I've made sure that all tokenizers have a new <|audio|> token to allow for better tracking audio placeholder tokens. This is only available on the tokenizer and not the embedding layer. As such, I intercept the input_ids before calling embedding on them and apply safe_input_ids instead.

When using V0, Ultravox has been verified to work on the following backends on an earlier version of this PR: Llama 3, Gemma 3, and Llama 4.

V0 seems to work as verified by evals. I've seen issues on V1 on an earlier version of VLLM, but I'm not sure if that was due to Ultravox or a VLLM V1 bug.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

What issue are you getting on V1?

[farzadab]

@DarkLight1337 When using V1, I noticed that the output was also completely garbled.

After debugging I noticed that when I tried printing input_ids here for the same sample (conditioned on len(input_ids)>1 to avoid decoding tokens), this is what I got:

# with VLLM_USE_V1=0
>>> t.decode([200000, 200005, 15651, 200006, 368, 4662, 583, 262, 19933, 43910, 26, 200008, 200005, 1556, 200006, 368, 4984, 290, 2182, 4097, 38, 7283, 201133, 200008, 200005, 140680, 200006, 368])
'<|begin_of_text|><|header_start|>system<|header_end|>\n\nYou are a helpful assistant.<|eot|><|header_start|>user<|header_end|>\n\nAnswer the following question: \n\n<|vision_reserved_special_token_1047|><|eot|><|header_start|>assistant<|header_end|>\n\n'

# with VLLM_USE_V1=1
>>> t.decode([24, 4984, 290, 2182, 4097, 38, 7283, 201133, 200008, 200005, 140680, 200006, 368])
',Answer the following question: \n\n<|vision_reserved_special_token_1047|><|eot|><|header_start|>assistant<|header_end|>\n\n'

The input_ids in the case of V1 seemed to be missing a part of the beginning.

[farzadab]

I believe I got this issue at around v0.8.4 or 0.8.4. I'll try verifying it on v0.8.5.post1.

[liPatrick]

Resolved by upgrading to v0.9.1

[liPatrick]

Verified that the issue with inference mismatch was indeed a VLLM bug. Upgrading to v0.9.1 fixed the issue and now V1 inference matches V0.

[DarkLight1337]

Nice, let's merge this!

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @farzadab.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @farzadab.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Blaze-DSP]

Hi!! I am trying to deploy fixie-ai/ultravox-v0_6-gemma-3-27b using vllm (vllm/vllm-openai:v0.9.2) in k8s.

- name: vllm-openai
          image: vllm/vllm-openai:v0.9.2
          imagePullPolicy: IfNotPresent
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: hf_token
            - name: HF_HUB_CACHE
              value: /mnt/models
          command: ["/bin/sh", "-c"]
          args:
            - |
              pip install "vllm[audio]" \
                && python3 -m vllm.entrypoints.openai.api_server \
                    --host 0.0.0.0 \
                    --port 8000 \
                    --uvicorn-log-level warning \
                    --model fixie-ai/ultravox-v0_6-gemma-3-27b \
                    --served-model-name ultravox-27b \
                    --device auto \
                    --trust-remote-code \
                    --max-model-len 4096 \
                    --enable-prefix-caching 

But, the following error occurs regarding Gemma3 vocab_size. Can anyone help? Saw a related PR #14687 but it didn't help

INFO 07-23 03:42:46 [__init__.py:244] Automatically detected platform cuda.
INFO 07-23 03:42:51 [api_server.py:1395] vLLM API server version 0.9.2
INFO 07-23 03:42:51 [cli_args.py:325] non-default args: {'host': '0.0.0.0', 'uvicorn_log_level': 'warning', 'model': '/mnt/models/ultravox-27b', 'trust_remote_code': True, 'max_model_len': 4096, 'served_model_name': ['ultravox-27b'], 'enable_prefix_caching': True}
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1495, in <module>
    uvloop.run(run_server(args))
  File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1431, in run_server
    await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1451, in run_server_worker
    async with build_async_engine_client(args, client_config) as engine_client:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 158, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 180, in build_async_engine_client_from_engine_args
    vllm_config = engine_args.create_engine_config(usage_context=usage_context)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1067, in create_engine_config
    model_config = self.create_model_config()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 956, in create_model_config
    return ModelConfig(
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/pydantic/_internal/_dataclasses.py", line 123, in __init__
    s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 533, in __post_init__
    hf_config = get_config(self.hf_config_path or self.model,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/config.py", line 361, in get_config
    config = config_class.from_pretrained(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 586, in from_pretrained
    return cls.from_dict(config_dict, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 746, in from_dict
    config = cls(**config_dict)
             ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/configs/ultravox.py", line 104, in __init__
    self.vocab_size = self.text_config.vocab_size
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 209, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Gemma3Config' object has no attribute 'vocab_size'```

[liPatrick]

Looks like adding this leads to CI errors #19580

[DarkLight1337]

cc @ywang96

[ywang96]

Yea this is expected because on huggingface there's no such model for us to test loading this particular model architecture. I wonder if we should relax this test on CI

[DarkLight1337]

Let's just add is_available_offline=False to disable the test for that model

[liPatrick]

Okay, I added back Llama4ForCausalLM to vllm/model_executor/models/registry.py and added Llama4ForCausalLM with is_available_online=False entry to tests/models/registry.py

[DarkLight1337]

The PP test failure looks related