Add Apertus #39381
Merged
Add Apertus #39381
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
EduardDurech
force-pushed
the
model/apertus
branch
from
1ea6373 to
353c6c0
Compare
EduardDurech
force-pushed
the
model/apertus
branch
from
1f20c58 to
1f4e715
Compare
|
@Cyrilvallez - this is the part 1 of the PR from Swiss AI initiative |
EduardDurech
force-pushed
the
model/apertus
branch
from
2728d3c to
b53417c
Compare
Yep, I was planning to and Cyril suggested, I built from Alex's original implementation but I'll refactor |
|
It should not require too much changes don't worry, its already in an excellent state! |
Cyrilvallez
reviewed
Aug 27, 2025
Co-authored-by: Cyril Vallez <[email protected]>
EduardDurech
force-pushed
the
model/apertus
branch
from
c66e7b4 to
792b7de
Compare
As the tests are written for a config without default scaling (which is not the case in Apertus) - besides, rope scaling is tested in other models so it's all safe.
Not needed (for now)
Following this: huggingface#39782
|
[For maintainers] Suggested jobs to run (before merge) run-slow: apertus, auto |
Cyrilvallez
approved these changes
Aug 28, 2025
4 tasks
4 tasks
|
Hey @EduardDurech do you think we can add integration tests now? 🤗 |
|
@ArthurZucker yea should be able, the models are hosted on HF now, @dhia680 would you be able to? I'm too busy with RL for a bit |
|
Nice! 🤗 we can also have a god if neither of you can!! |
vermouth1992
pushed a commit
to volcengine/verl
that referenced
this pull request
Sep 13, 2025
Pre-release of Apertus from the Swiss AI Initiative Main modifications from Llama - xIELU Activation - QK-norm Associated Transformers PR huggingface/transformers#39381 Associated vLLM PR vllm-project/vllm#23068 Associated SGLang PR sgl-project/sglang#9774 GSM8K <img width="430" height="262" alt="image" src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed" /> <img width="436" height="266" alt="image" src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08" />
wlf-darkmatter
pushed a commit
to wlf-darkmatter/verl
that referenced
this pull request
Sep 13, 2025
Pre-release of Apertus from the Swiss AI Initiative Main modifications from Llama - xIELU Activation - QK-norm Associated Transformers PR huggingface/transformers#39381 Associated vLLM PR vllm-project/vllm#23068 Associated SGLang PR sgl-project/sglang#9774 GSM8K <img width="430" height="262" alt="image" src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed" /> <img width="436" height="266" alt="image" src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08" />
|
@andresnowak has a draft PR for tests #41037 if you guys want to check, in the meantime there are xIELU CUDA parity issues (already known issue) I asked the group about, will see if that's fixed before and included |
VocabVictor
pushed a commit
to VocabVictor/verl-plus
that referenced
this pull request
Sep 24, 2025
Pre-release of Apertus from the Swiss AI Initiative Main modifications from Llama - xIELU Activation - QK-norm Associated Transformers PR huggingface/transformers#39381 Associated vLLM PR vllm-project/vllm#23068 Associated SGLang PR sgl-project/sglang#9774 GSM8K <img width="430" height="262" alt="image" src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed" /> <img width="436" height="266" alt="image" src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08" />
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pre-release of Apertus from the Swiss AI Initiative
Main modifications from Llama
@ArthurZucker