-
Notifications
You must be signed in to change notification settings - Fork 118
Add SmolLM3 #422
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add SmolLM3 #422
Conversation
jonatanklosko
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @joelpaulkoch, this looks great! I dropped a few small comments and it's good to go :)
|
The implementation is basically llama + NoPE support (in the transformer block) + architectures that are supported but missing in llama (i.e. |
It's separate in hf/transformers, so I would keep it separate here to for consistency. Also, I wouldn't necessarily add features to llama that are not in the hf/transformers implementation, otherwise it's harder to analyse for parity :) |
test/bumblebee/text/smollm3_test.exs
Outdated
| atol: 1.0e-3, | ||
| rtol: 1.0e-3 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is unusual, pretty much all other LLMs work with the default atol 10e-4. If the numbers are slightly off like that, it often indicates a small difference, like a missing layer norm, layer norm in a different order, or something like that.
Do you know if that deviation is only for the test models, or is it similar for any real checkpoint?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I checked and it's the same for HuggingFaceTB/SmolLM3-3B, not sure what is missing.
A missing layer norm would show up in the debug logs, right?
I'll investigate further when I find the time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A missing layer norm would show up in the debug logs, right?
A norm layer that bumblebee has and cannot be found in the checkpoint would be logged. But if there is a norm layer in the checkpoint that bumblebee doesn't use, then it is not logged, unless we pass log_params_diff: true to Bumblebee.load_model (sometimes unused layers are expected, for example if you load a checkpoint as a base model, ignoring the head layers). But I just tried with log_params_diff: true and it doesn't log anything, so in terms of layers it looks like they match.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@joelpaulkoch I had a look and found the issue, see bf4d8d8. I first checked the config loaded from HF and noticed double nesting in rotary_embedding_enabled: %{rotary_embedding_enabled: ...}. And the main issue was reading :rotary_embedding_enabled from opts, rather than spec, which means it was never actually applied:
- case opts[:rotary_embedding_enabled] do
+ case spec.rotary_embedding_enabled doBtw. Looking at this, I realised that that rotary embedding layer doesn't have any params, so we don't need the layer mapping:
{"decoder.blocks.#{index}.self_attention.rotary_embedding",
"model.layers.#{index}.self_attn.rotary_emb"}I believe it was added as part of llama unnecessarily, and then by copying we kept that across many models, but it doesn't actually do anything. I will remove this mapping everywhere in a separate commit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh great, thank you @jonatanklosko !!
jonatanklosko
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!
Hey, this is the SmolLM3 model from huggingface. It's smol, fully open and supports reasoning, so I figured it would be a nice addition to bumblebee.
I didn't implement YaRN extrapolation.