[bug] `facebook/wav2vec2-conformer-rope-large-960h-ft` refuses to work in `fp16`

[Vaibhavs10]

System Info

• transformers version: 4.32.1
• Platform: Linux-5.15.109+-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.16.4
• Safetensors version: 0.3.3
• Accelerate version: not installed
• Accelerate config: not found
• PyTorch version (GPU?): 2.0.1+cu118 (True)
• Tensorflow version (GPU?): 2.12.0 (True)
• Flax version (CPU?/GPU?/TPU?): 0.7.2 (gpu)
• Jax version: 0.4.14
• JaxLib version: 0.4.14
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help?

@sanchit-gandhi @patrick

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

facebook/wav2vec2-conformer-rope-large-960h-ft throws a rather cryptic error (RuntimeError: mat1 and mat2 must have the same dtype) when loaded in half-precision.
repro: https://github.com/Vaibhavs10/scratchpad/blob/main/conformer_wav2vec2_repro.ipynb
model: https://huggingface.co/facebook/wav2vec2-conformer-rope-large-960h-ft
Note: It works fine on fp32

Expected behavior

It should work without any issues!