[speech2text] fix init of sinusoidal embeddings #37931
Conversation
|
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the |
| from datasets import load_dataset | ||
|
|
||
| @classmethod | ||
| def setUpClass(cls): |
There was a problem hiding this comment.
prevents regressions: the model here is super small, these 2 tests take <10s on CPU
most of the time is in loading the model and dataset, having setUpClass cuts the run time from 15 to <10s
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| module.weight.data.normal_(mean=0.0, std=std) | ||
| if module.padding_idx is not None: | ||
| module.weight.data[module.padding_idx].zero_() | ||
| elif isinstance(module, MusicgenSinusoidalPositionalEmbedding): |
There was a problem hiding this comment.
(we were initializing these weights buffers twice)
| self.assertTrue( | ||
| output_values.shape == (2, 1, 36480) | ||
| ) # input values take shape 32000 and we generate from there | ||
| torch.testing.assert_close(output_values[0, 0, -16:].cpu(), EXPECTED_VALUES, rtol=1e-4, atol=1e-4) |
There was a problem hiding this comment.
slow ci becomes green with this tol update
* fix init (meta device -> bad numbers) * fast test * dont init sinusoidal twice * make fixup
3 participants
What does this PR do?
Fixes #37874
Problem:
nn.Parameter()defined in__init__-> we initialize in themetadevice -> whatever is stored in this parameter at init time gets shreddedSolution:
Don't init the sinusoidal embeddings in
nn.Parameter, e.g. use a non-persistent buffer. The buffer solution is used in other models with sinusoidal embeddings, e.g. here