Skip to content

Gemma3 #36658

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 126 commits into from
Mar 12, 2025
Merged

Gemma3 #36658

merged 126 commits into from
Mar 12, 2025
+5,469 −116

Conversation

@RyanMullins
Copy link
Contributor

What does this PR do?

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

xenova and others added 30 commits March 5, 2025 16:45
@xenova @RyanMullins
Fix converter
5faeae3
@RyanMullins
[Broken] Adds Gemma 3 to Hugging Face Transformers
b21634b
@RyanMullins
Consolidating Config and Processor params across impls
8e27cd7
@RyanMullins
Sorting out configuration parameters. Adds qk_norm before RoPE. Still…
72b60b0 
… not sure if RoPE is right.
@RyanMullins
Additional plumbing for CausalLM and ConditionalGeneration variants
e01056e
@RyanMullins
incomplete draft of Orbax conversion script
9a08450
@RyanMullins
More complete checkpoint conversion
a7d7fb2
@RyanMullins
Supporting Gemma 3 1B checkpoints
6d5b637
@RyanMullins
Updating RoPE for multiple frequencies
2699ec6
@RyanMullins
Adjustments to rotary embedder
49c8658
@RyanMullins
Proof of life for text-only operation
146822a
@RyanMullins
Updating the conversion script to handle multimodal projection weights
74f4acb
@RyanMullins
Fixing tet-only conversions
bfcc303
@RyanMullins
Cleaner conversion script with multimodal support and a simpler proce…
88897b2 
…ssor
@RyanMullins
Additional refatcors to the Gemma3Processor
0548c26
@RyanMullins
Simplified Processor to work over text representations
1a860c7
@RyanMullins
Updated conversion script to join text and vision embeddings at conve…
f9036cd 
…rion time
@RyanMullins
Logging for debugging
61f0b58
@RyanMullins @xenova
Update src/transformers/models/gemma2/modeling_gemma2.py
3f282c9 
Co-authored-by: Joshua Lochner <[email protected]>
@RyanMullins
Removed extraneous Config params
8b41347
@RyanMullins
Switching to fast tokenizer for checkpoint conversions
daacc1d
@RyanMullins
isolating siglip for performance tetsing
4338957
@RyanMullins
Minor changes for debugging tests against baselines
14c443c
@RyanMullins
Adding average pooling for soft tokens
d45be31
@RyanMullins
Updating processor code to enable simpler embedding interleaving for …
cdbd03f 
…arbitrary number of images in prompts
@RyanMullins
Updating conversion script for ShieldGemma 2 conversion compatibility
ec2a7df
@pcuenca @RyanMullins
Allow disable_compile to be provided as a kwarg
85d1181
@pcuenca @RyanMullins
Refresh from modular
6922438
@RyanMullins
Updated conversion script and corrected sliding window
f47afe2
@pcuenca @RyanMullins
Fix type mismatch in cache_position (huggingface#4)
c40f6e2
RyanMullins and others added 21 commits March 11, 2025 14:16
@RyanMullins
Inverting prior BOS change
8ab84bb
@zucchini-nlp
ultra nit
6dd1aef
@RyanMullins
Reverting to Tokenizer saved with add_bos_token=True and chat templat…
ae80685 
…e starting with BOS
@zucchini-nlp
resize embeds, remove sqrt, add slow test outputs
ba77bc5
@zucchini-nlp
FA2 but quality is meh
aa9d141
@zucchini-nlp
Merge pull request huggingface#9 from huggingface/raushan-working
35ff071 
Raushan address PR comments
@zucchini-nlp
nit
ca82ebc
@zucchini-nlp
skip FA2, no idea what happened
74da721
@zucchini-nlp
last bit for green CI
123402a
@zucchini-nlp
please, green CI for docs
d541fe4
@zucchini-nlp
T_T
1280714
@RyanMullins
Fix for Gemma3 logits
4914133
@xenova
Support both options for system prompt
4c48f13
@xenova
Add support for both forms of system prompts
1711942
@RyanMullins @pcuenca
Update src/transformers/models/gemma3/image_processing_gemma3_fast.py
5ad5b27 
Co-authored-by: Pedro Cuenca <[email protected]>
@RyanMullins @pcuenca
Update docs/source/en/model_doc/gemma3.md
c3b0213 
Co-authored-by: Pedro Cuenca <[email protected]>
@RyanMullins @pcuenca
Update docs/source/en/model_doc/gemma3.md
2dd948b 
Co-authored-by: Pedro Cuenca <[email protected]>
@RyanMullins @pcuenca
Update docs/source/en/model_doc/gemma3.md
5f8f8a6 
Co-authored-by: Pedro Cuenca <[email protected]>
@RyanMullins @pcuenca
Update docs/source/en/model_doc/gemma3.md
cd14f3f 
Co-authored-by: Pedro Cuenca <[email protected]>
@RyanMullins @pcuenca
Update docs/source/en/model_doc/gemma3.md
782bb92 
Co-authored-by: Pedro Cuenca <[email protected]>
@RyanMullins
Docs updates now that assets are live
a334121
@github-actions github-actions bot marked this pull request as draft March 12, 2025 07:36
@github-actions
Copy link
Contributor

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the Ready for review button (at the bottom of the PR page).

ArthurZucker
ArthurZucker approved these changes Mar 12, 2025
View reviewed changes
Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As reviewed before LGTM!

@LysandreJik LysandreJik marked this pull request as ready for review March 12, 2025 07:40
@LysandreJik LysandreJik merged commit 50d3530 into huggingface:main Mar 12, 2025
17 of 21 checks passed
@NanoCode012 NanoCode012 mentioned this pull request Mar 12, 2025
Merged
DarkLight1337
DarkLight1337 reviewed Mar 12, 2025
View reviewed changes
src/transformers/models/auto/modeling_auto.py
("fnet", "FNetForPreTraining"),
("fsmt", "FSMTForConditionalGeneration"),
("funnel", "FunnelForPreTraining"),
("gemma3", "Gemma3ForConditionalGeneration"),
Copy link

@DarkLight1337 DarkLight1337 Mar 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be included in MODEL_FOR_VISION_2_SEQ_MAPPING_NAMES as well? I can't load the multimodal variant using AutoModelForVision2Seq, which works for most multimodal models.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VLMs should be loaded with AutoModelForImageTextToText which is a new mapping we added for multimodal models. The old AutoModelForVision2Seq is supposed to work only for models like BLIP which are used to take bare images without instructions, and caption them

Since earlier we didn't have a specific mapping for VLMs, everything got dumped in Vision2Seq, sorry if it was confusing. New releases all will come under ImageTextToText + all older models support this mapping

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thanks for the explanation!

@DarkLight1337 DarkLight1337 mentioned this pull request Mar 17, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zucchini-nlp zucchini-nlp zucchini-nlp left review comments

@ArthurZucker ArthurZucker ArthurZucker approved these changes

+1 more reviewer

@DarkLight1337 DarkLight1337 DarkLight1337 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@RyanMullins @DarkLight1337 @ArthurZucker @zucchini-nlp @LysandreJik @xenova @pcuenca @MayankChaturvedi