MptForCausalLM.from_pretrained gives error 'dict' object has no attribute 'softmax_scale'

[abacaj]

System Info

Creating model MPT:

    model = MptForCausalLM.from_pretrained(
        model_args.model_name_or_path,
        cache_dir=training_args.cache_dir,
        torch_dtype=torch.bfloat16,
        use_cache=False,
        init_device=f"cuda:{local_rank}",
        attn_config=dict(attn_impl="flash", softmax_scale=None),  # triton, flash
    )

Gives the following error:

  File "/home/anton/personal/stanford_alpaca-replit/env/lib/python3.10/site-packages/transformers/models/mpt/modeling_mpt.py", line 258, in __init__
    self.attn = MptAttention(config)
  File "/home/anton/personal/stanford_alpaca-replit/env/lib/python3.10/site-packages/transformers/models/mpt/modeling_mpt.py", line 137, in __init__
    self.softmax_scale = config.attn_config.softmax_scale
AttributeError: 'dict' object has no attribute 'softmax_scale'

Who can help?

Information

Tasks

Reproduction

    model = MptForCausalLM.from_pretrained(
        model_args.model_name_or_path,
        cache_dir=training_args.cache_dir,
        torch_dtype=torch.bfloat16,
        use_cache=False,
        init_device=f"cuda:{local_rank}",
        attn_config=dict(attn_impl="flash", softmax_scale=None),  # triton, flash
    )

Expected behavior

Should be using MptAttentionConfig instead is using a dict object

[sgugger]

cc @younesbelkada and @ArthurZucker

[ArthurZucker]

Hey! Thanks for reporting. Indeed this does not work as the attn_config object does not seem to get converted to a MptAttentionConfig object. The MptAttention still needs the full config.
Will open a PR to fix this as I guess this was previously working!

[ArthurZucker]

It is not something that we usually support. For composition models like CLIP, CLAP etc, this would not work:

from transformers import CLIPModel
CLIPModel.from_pretrained("openai/clip-vit-base-patch16", text_config = dict(num_hidden_layers = 2))
....
│ /home/arthur_huggingface_co/transformers/src/transformers/models/clip/configuration_clip.py:411  │
│ in to_dict                                                                                       │
│                                                                                                  │
│   408 │   │   │   `Dict[str, any]`: Dictionary of all the attributes that make up this configu   │
│   409 │   │   """                                                                                │
│   410 │   │   output = copy.deepcopy(self.__dict__)                                              │
│ ❱ 411 │   │   output["text_config"] = self.text_config.to_dict()                                 │
│   412 │   │   output["vision_config"] = self.vision_config.to_dict()                             │
│   413 │   │   output["model_type"] = self.__class__.model_type                                   │
│   414 │   │   return output                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'dict' object has no attribute 'to_dict'

We allow:

from transformers import CLIPModel, CLIPTextConfig
text_config = CLIPTextConfig(num_hidden_layers = 2)
CLIPModel.from_pretrained("openai/clip-vit-base-patch16", text_config = text_config)

However for backward compatibility, #25116 will fix this for MPT