hidden_dropout_prob and attention_probs_dropout_prob values in the documentation doesn't match with that in the code

[SwapnanilMukherjee]

System Info

• transformers version: 4.30.2
• Platform: Linux-5.15.0-79-generic-x86_64-with-glibc2.29
• Python version: 3.8.10
• Huggingface_hub version: 0.16.2
• Safetensors version: 0.3.1
• PyTorch version (GPU?): 2.0.1+cu117 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: RTX A5000
• Using distributed or parallel set-up in script?: No

Who can help?

@sgugger

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I am training ViLT on e-SNLI-VE. This is basically what I am doing.

from transformers import ViltConfig, ViltProcessor, ViltForQuestionAnswering

label2id = {'contradiction':0, 'entailment':1, 'neutral':2}
id2label = {0:'contradiction', 1:'entailment', 2:'neutral'}

vilt_config = ViltConfig(label2id=label2id, id2label=id2label, max_position_embeddings=100)
vilt_config

This gives the following output.

The values for both variables should be 0.1 as indicated by the documentation.

Expected behavior

The values of the variables should have been zero.

[stevhliu]

Thanks for catching the typo, would you like to open a PR to change the values to 0.0? 🤗

[SwapnanilMukherjee]

Yes, I'll do that right away. But should the value in the documentation or the model init be changed?

[stevhliu]

I think the value should be changed in the docstring to reflect whats in the init (pinging @ArthurZucker for confirmation):

transformers/src/transformers/models/vilt/configuration_vilt.py

Line 112 in 41aef33

hidden_dropout_prob=0.0,

[ArthurZucker]

Values in both the documentation and the default value should match that of https://huggingface.co/dandelin/vilt-b32-mlm 😉

[SwapnanilMukherjee]

The value of dropout in the original implementation by dandelin is 0.1 . Since the value in the docstring is already that, I'm changing the init to that and creating a PR.