Mismatching default value of `Llama4TextConfig` `attn_temperature_tuning` between official llama code

[gmlwns2000]

System Info

In transfomers library, the default value of attn_temperature_tuning in Llama4TextConfig, is 4.

transformers/src/transformers/models/llama4/configuration_llama4.py

Line 231 in 953196a

attn_temperature_tuning (`int`, *optional*, defaults to 4): TODO

However, I think this value should be boolean because it is used as a condition flag in the forward pass.

transformers/src/transformers/models/llama4/modeling_llama4.py

Line 334 in 953196a

if self.attn_temperature_tuning and not self.use_rope:

Moreover, in the official implementation, that value is boolean, which is defaulted to False. (Offical Config File)

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Any Llama4TextConfig instances.

Expected behavior

The default value of config.attn_temperature_tuning should be True or False depending on input sequence length.

[Rocketknight1]

Searched the codebase and the value is indeed only used as a boolean rather than an int - cc @ArthurZucker should we just make it True/False?

[gmlwns2000]

Hi, I will make a PR of it if we need to change it to a boolean.

[Rocketknight1]

Arthur is out actually! @Cyrilvallez are you okay with the change? I'm pretty sure it's correct

[Cyrilvallez]

Indeed, this is definitely a mismatch! Will be much clearer as a bool! All green if you want to open a PR to correct it @gmlwns2000! 🤗

[gmlwns2000]

I just made the PR that fixes this issue!