Skip to content

Conversation

@vaibhavjindal
Copy link
Collaborator

Summary

Fixes #739.

Creates new classes for various RMS norms and removes the use of partial for RMS norms.

Testing Done

Fixes errors of the form:

TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union

for glm4, olmo2, gemma1 and gemma2.

However, we are now seeing errors when matching the actual logits with gemma models, which can be tracked separately in #729.

  • Hardware Type:
  • run make test to ensure correctness
  • run make checkstyle to ensure code style
  • run make test-convergence to ensure convergence

@vaibhavjindal vaibhavjindal marked this pull request as ready for review June 2, 2025 09:55
@vaibhavjindal vaibhavjindal mentioned this pull request Jun 2, 2025
17 tasks
Copy link
Collaborator

@Tcc0403 Tcc0403 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@Tcc0403 Tcc0403 merged commit 4a2da99 into main Jun 2, 2025
3 of 7 checks passed
@Tcc0403 Tcc0403 deleted the fix-rmsnorm-patch branch June 2, 2025 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

patching LigerRMSNorm with partial arguments given would cause error in _init_weights()

3 participants