[RFC]: Decoupling vLLM Configuration from Hugging Face

[charlotte12l]

Motivation.

Currently, vLLM assumes that all models follow the Hugging Face format. Configuration is parsed directly into a transformers.PretrainedConfig instance, which is then embedded into ModelConfig as hf_config. This tight coupling introduces several problems:

• Poor extensibility: non-HF models (e.g., Mistral-native) cannot be integrated cleanly. Their configuration must first be awkwardly adapted into a PretrainedConfig-like object.

• Maintenance overhead: many fields in PretrainedConfig are irrelevant to inference, but vLLM does not clearly separate used vs. unused fields. Users who want to support their own model formats must carefully map into HF’s schema, and this kind of manual mapping is fragile and error-prone.

• Under‑specified, inconsistently named critical fields[added on Nov 1]: Fields vLLM relies on at runtime—such as num_kv_heads, num_experts, max_model_len—are not emphasized or standardized in PretrainedConfig. Different models use different names, forcing ModelConfig to perform bespoke mappings.

• Missing architecture hints force runtime introspection[added on Nov 1]: Useful architecture details (e.g., attention type per layer for KV‑cache initialization) are not guaranteed(maybe it’s now? but what if there are other hints needed in the future?) available in PretrainedConfig. Today we infer them at runtime (e.g., via forward_context), adding complexity and risk. Getting such fields into HF first and then into every model slows vLLM development.

This proposal aims to resolve these issues by introducing a clean separation of concerns:

• Define a standardized, vLLM‑native configuration schema that clearly defines the fields the engine needs.
• Provide pluggable parsers that translate external configuration formats (HF, Mistral‑native, GGUF, etc.) into that schema.
• Simplify code paths: fewer runtime probes (e.g., no per‑layer attention discovery via forward_context) and earlier detection of missing information.

Proposed Change.

1. Unified Configuration Schema

Introduce a new class, tentatively named ModelArchitectureConfig (or a better name lol) , that contains only the essential fields required by vLLM for inference:

class ModelArchitectureConfig:
    architectures: List[str]
    model_type:str
    hidden_size: int
    num_hidden_layers: int
    num_attention_heads: int
    head_dim: int  
    vocab_size: int

def __init__(self,    
   architectures,
   model_type,
   hidden_size,
   num_hidden_layers,
   num_attention_heads,
   head_dim,
   vocab_size,
   **kwargs,
):
   ...
      for key, value in kwargs.items():
           setattr(self, key, value)


def validate():

Current structure:

VLLMConfig
 └── ModelConfig
       └── hf_config: PretrainedConfig

Proposed structure:

VLLMConfig
 └── ModelConfig
       ├── unified_config: UnifiedPretrainedConfig   # always present
       └── hf_config: PretrainedConfig    # optional

• Engine logic consumes only unified_config.
• hf_config is retained when loading from Hugging Face, but becomes optional.

2. Parsers

With ModelArchitectureConfig , each model format can be supported through a dedicated parser. Parsers always ensure the vllm-runtime required fields will be correctly extracted, while leaving the rest model specific fields as it is.

Similar to #24277, we can do:

class HFConfigParser(ConfigParserBase):

    def parse(self,
              model: Union[str, Path],
              trust_remote_code: bool,
              revision: Optional[str] = None,
              code_revision: Optional[str] = None,
              **kwargs) -> UnifiedPretrainedConfig:

@register_config_parser("custom_config_parser")
class CustomConfigParser(ConfigParserBase):

    def parse(self,
              model: Union[str, Path],
              trust_remote_code: bool,
              revision: Optional[str] = None,
              code_revision: Optional[str] = None,
              **kwargs) -> UnifiedPretrainedConfig:
        raise NotImplementedError

3. Engine surface change

Where the engine previously reached for model_config.hf_config, switch to the model_config.unified_pretrained_config:

4. Migration Plan (updated on Nov 1)

A. Add ModelArchitectureConfig interface, HFArchitectureConfigParser and MistralArchitectureConfigParser to parse their trained model params into ModelArchitectureConfig
B. Update ModelConfig to include model_architecture_config and make hf_config optional.
C. During kv cache initialization, call model_architecture_config.layer_attention_types to get each layer’s attention type. Also refactor get_kv_cache_spec a bit to make it able to get spec from a un-initialized module
D. Calls to hf_config during vllm-runtime are gradually replaced with model_architecture_config.
E. Calls to hf_config in model_executor/models/*.py are gradually replaced with model_architecture_config.
F. Keep hf_config but mark it as deprecated, populate it only if the source was a hugging face model
G. Completely remove hf_config

Feedback Period.

1-2 weeks

CC List.

@22quinn @zhuohan123 @yeqcharlotte @houseroad @simon-mo

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[22quinn]

cc @hmellor @WoosukKwon

[hmellor]

Before I start, I want to make it clear that I totally understand the benefits of this for private/proprietary OOT checkpoint formats and am not against enabling this use case.

---

Poor extensibility: non-HF models (e.g., Mistral-native) cannot be integrated cleanly. Their configuration must first be awkwardly adapted into a PretrainedConfig-like object.

As far as I know, Mistral models are the only models to do this, and their checkpoints also contain HF format configs. This should make Mistral format configs unnecessary in vLLM.

Are there other cases that I'm not aware of?

Maintenance overhead: many fields in PretrainedConfig are irrelevant to inference, but vLLM does not clearly separate used vs. unused fields. Users who want to support their own model formats must carefully map into HF’s schema, which is fragile and error-prone.

This may be a naive question, but why are unused fields a problem?

Regarding the fragility of HF's schema, could you please elaborate on what you mean by that? I'm not sure I understand.

---

If we can resolve these issues on the Transformers side these pain points from the integration with vLLM can be used to benefit the entire community of Transformers users.

[charlotte12l]

@hmellor Thanks for the questions.

As far as I know, Mistral models are the only models to do this, and their checkpoints also contain HF format configs. This should make Mistral format configs unnecessary in vLLM.
Are there other cases that I'm not aware of?

Yes, for in-tree models, only Mistral. But there are out-of-tree models that do not use HF format and need to do the conversion.

This may be a naive question, but why are unused fields a problem?
Regarding the fragility of HF's schema, could you please elaborate on what you mean by that? I'm not sure I understand.

That's also for users who are going to support non-HF format models. Because of those unused fields, users who want to support their own model formats must carefully map into HF’s schema. I didn't mean HF's schema is fragile, I meant the thing "users needing to carefully mapping into HF's schema" is fragile and error-prone.

[zhuohan123]

Some questions/suggestions:

1. Let's call UnifiedPretrainedConfig something like ArchitectureConfig or ModelArchitectureConfig? Since this is the main info it includes.
2. Are these the only fields we use in vLLM Core part (like in engine, basically any part outside of the model code)?

    architectures: List[str]
    model_type:str
    hidden_size: int
    num_hidden_layers: int
    num_attention_heads: int
    head_dim: int  
    vocab_size: int

[hmellor]

Yes, for in-tree models, only Mistral. But there are out-of-tree models that do not use HF format and need to do the conversion.

I agree that this makes sense for out-of-tree private/proprietary models.

However, are there actually any open-source out-of-tree models?

That's also for users who are going to support non-HF format models. Because of those unused fields, users who want to support their own model formats must carefully map into HF’s schema. I didn't mean HF's schema is fragile, I meant the thing "users needing to carefully mapping into HF's schema" is fragile and error-prone.

Thank you for clarifying. So you're saying that it's not clear which fields vLLM is using, so users don't know which fields they should be using when defining their config in HF format?

[ywang96]

I totally understand the need for a unified config to support non-HF format, and I see there are two parts of this problem:

1. UnifiedPretrainedConfigt to allow model configuration loading from non-hf standard - I actually very much agree with this and I think even some of the hf key naming are outdated/confusing due to legacy reasons (e.g., max_position_embeddings)
2. Update ModelConfig to include unified_pretrained_config and make hf_config optional -> Completely remove hf_config - I'm personally not too sure about with this. Almost all models supported by vLLM today are released as a HF model first. Does this mean the model vendor need to maintain this parsing logic, or vLLM needs to maintain it? This also means we need to replace hf_config everywhere in the codebase.

On a side note, how would we manage loading tokenizer and/or processor?

[hmellor]

some of the hf key naming are outdated/confusing due to legacy reasons (e.g., max_position_embeddings)

HF key naming is 100% updatable. We are not limited to what is present in the config.json of old checkpoints. If there is a better name for a config, we can remap it in Transformers and vLLM can just read the new name.

[patrickvonplaten]

As far as I know, Mistral models are the only models to do this, and their checkpoints also contain HF format configs. This should make Mistral format configs unnecessary in vLLM.

Just a quick side-note here since Mistral models are mentioned.

We don't have HF format configs for all mistral models - e.g.: https://huggingface.co/mistralai/Pixtral-Large-Instruct-2411 does not have it.

We have it for most, but it should never be the source of truth. The source of truth is always --config_format mistral and params.json.

Generally it ties in a bit with the question to what extend vLLM should be tied to transformers. I think it's amazing that vLLM can automatically leverage all transformers models, but it's also important to allow people to side step transformers in case the model is not implemented there no?

This would include:

• A custom config
• Skip tokenizer (think that's already done here: [Bugfix] Allow --skip-tokenizer-init with echo and return_token_ids #26238)
• Also a custom nn.Module architecture where vllm's amazing attn kernels can be plugged in easily

It'd be amazing to also a "barebone" vLLM option in the future no that essentially can just run on a nn.Module as it'd enable a bunch more use cases

[charlotte12l]

Some questions/suggestions:

1. Let's call UnifiedPretrainedConfig something like ArchitectureConfig or ModelArchitectureConfig? Since this is the main info it includes.
2. Are these the only fields we use in vLLM Core part (like in engine, basically any part outside of the model code)?

    architectures: List[str]
    model_type:str
    hidden_size: int
    num_hidden_layers: int
    num_attention_heads: int
    head_dim: int  
    vocab_size: int

@zhuohan123 Make a more detailed investigation, the below ones should be good enough

class ModelArchitectureConfig:
    architectures: List[str]
    model_type:str
    # Below are vllm runtime required fields
    hidden_size: int
    num_hidden_layers: int
    num_attention_heads: int
    head_dim: int  
    vocab_size: int
    num_key_value_heads: int 
    num_experts: int 
    max_model_len: int 
    layer_attention_types: List[str] 

[charlotte12l]

Yes, for in-tree models, only Mistral. But there are out-of-tree models that do not use HF format and need to do the conversion.

I agree that this makes sense for out-of-tree private/proprietary models.

However, are there actually any open-source out-of-tree models?

That's also for users who are going to support non-HF format models. Because of those unused fields, users who want to support their own model formats must carefully map into HF’s schema. I didn't mean HF's schema is fragile, I meant the thing "users needing to carefully mapping into HF's schema" is fragile and error-prone.

Thank you for clarifying. So you're saying that it's not clear which fields vLLM is using, so users don't know which fields they should be using when defining their config in HF format?

@hmellor
Without a clear specification, users might be confused which fields or naming conventions are required by vLLM at the engine level, e.g. Some critical fields like num_kv_heads, num_experts, max_model_len—are not emphasized or standardized in PretrainedConfig. This leads to two main concerns:

• Different models may use different field names, requiring vLLM’s ModelConfig to handle custom mappings for each case.
• Users onboarding their own models to vLLM might not be aware of these critical fields, or may use inconsistent naming. They might work around this by adding more [possible_keys] (

  vllm/vllm/config/model.py

  Lines 2028 to 2056 in b5d90f7

  	def _get_and_verify_max_len(
  	hf_config: PretrainedConfig,
  	tokenizer_config: dict | None,
  	max_model_len: int | None,
  	disable_sliding_window: bool,
  	sliding_window: int | None,
  	spec_target_max_model_len: int | None = None,
  	encoder_config: Any | None = None,
  	) -> int:
  	"""Get and verify the model's maximum length."""
  	derived_max_model_len = float("inf")
  	possible_keys = [
  	# OPT
  	"max_position_embeddings",
  	# GPT-2
  	"n_positions",
  	# MPT
  	"max_seq_len",
  	# ChatGLM2
  	"seq_length",
  	# Command-R
  	"model_max_length",
  	# Whisper
  	"max_target_positions",
  	# Others
  	"max_sequence_length",
  	"max_seq_length",
  	"seq_len",
  	]

  ), which isn’t sustainable long-term.

Ideally, with the new configuration approach, users will have clear guidance on the critical fields required by vLLM. Any necessary name mapping should be handled in the parser, so that ModelConfig always accesses consistent field names, rather than relying on possible_keys

[charlotte12l]

I totally understand the need for a unified config to support non-HF format, and I see there are two parts of this problem:

1. UnifiedPretrainedConfigt to allow model configuration loading from non-hf standard - I actually very much agree with this and I think even some of the hf key naming are outdated/confusing due to legacy reasons (e.g., max_position_embeddings)
2. Update ModelConfig to include unified_pretrained_config and make hf_config optional -> Completely remove hf_config - I'm personally not too sure about with this. Almost all models supported by vLLM today are released as a HF model first. Does this mean the model vendor need to maintain this parsing logic, or vLLM needs to maintain it? This also means we need to replace hf_config everywhere in the codebase.

On a side note, how would we manage loading tokenizer and/or processor?

@ywang96

Good point! Regarding tokenizers, if I understand correctly: for Hugging Face models, we primarily need tokenizer_config.json, tokenizer.model, tokenizer.json . Only if tokenizer_class is not present in tokenizer_config.json, it will fall back to loading/checking huggingface config and access tokenizer_class there. https://github.com/huggingface/transformers/blob/v4.57.1/src/transformers/models/auto/tokenization_auto.py#L1076-L1086.

Given this, I think we can keep the tokenizer loading logic as it is—deprecating hf_config shouldn’t impact it. People can also use plugin to support customized tokenizer #12518, What do you think?

For loading processor, would appreciate your inputs as I'm not familiar the codes there. From what I can tell, it also seems fairly independent from hf_config/PretrainedConfig https://github.com/vllm-project/vllm/blob/main/vllm/transformers_utils/processor.py#L87 so deprecating hf_config/PretrainedConfig would be fine? For calls like

self.ctx.init_processor(xxxProcessor, config=self.get_hf_config(), ...)

it looks like the xxxProcessor are defined in vllm/model_executor/models/, so we can modify the processors init to accept ModelArchitectureConfig instead?
https://github.com/search?q=repo%3Avllm-project%2Fvllm+self.ctx.init_processor&type=code

Let me know if this makes sense, or if there are any edge cases I might have missed!

[heheda12345]

@charlotte12l Is it enough? I'm trying to list some in my mind:

1. whether this is a multi-modal model
2. what types of mamba layers this model includes and what's the state size
3. does this model use mla
4. what's the dtype of model weight
5. does this model support eplb

Some questions/suggestions:

1. Let's call UnifiedPretrainedConfig something like ArchitectureConfig or ModelArchitectureConfig? Since this is the main info it includes.
2. Are these the only fields we use in vLLM Core part (like in engine, basically any part outside of the model code)?

    architectures: List[str]
    model_type:str
    hidden_size: int
    num_hidden_layers: int
    num_attention_heads: int
    head_dim: int  
    vocab_size: int

@zhuohan123 Make a more detailed investigation, the below ones should be good enough

class ModelArchitectureConfig:
    architectures: List[str]
    model_type:str
    # Below are vllm runtime required fields
    hidden_size: int
    num_hidden_layers: int
    num_attention_heads: int
    head_dim: int  
    vocab_size: int
    num_key_value_heads: int 
    num_experts: int 
    max_model_len: int 
    layer_attention_types: List[str] 

[heheda12345]

I like this RFC. But how to make sure the migration is correct? And when we add a new attribute, how to ensure this attribute is parsed correctly for all existing models?