[ORPO] Move ORPOTrainer to experimental #4480
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Move ORPOTrainer and ORPOConfig to trl.experimental.orpo - Add deprecation warnings in trl.trainer with removal planned for TRL 0.29.0 - Update imports in tests, examples, and documentation - Maintain backward compatibility through deprecation stubs Fixes #4465
Member
|
nice, you'll need to replace all |
qgallouedec
reviewed
Nov 6, 2025
docs/source/orpo_trainer.md
Outdated
Comment on lines
124
to
131
| [[autodoc]] ORPOTrainer | ||
| - train | ||
| - save_model | ||
| - push_to_hub | ||
|
|
||
| ## ORPOConfig | ||
|
|
||
| [[autodoc]] ORPOConfig |
Member
There was a problem hiding this comment.
You need to update this as well
[[autodoc]] experimental.orpo.ORPOTrainer
[[autodoc]] experimental.orpo.ORPOConfig
qgallouedec
reviewed
Nov 6, 2025
Comment on lines
117
to
118
| from .orpo_config import ORPOConfig | ||
| from .orpo_trainer import ORPOTrainer |
Member
There was a problem hiding this comment.
you need to revert this change, otherwise from trl import ORPOTrainer won't work anymore
qgallouedec
reviewed
Nov 6, 2025
Member
There was a problem hiding this comment.
could you move this file to tests/experimental
trl/trainer/orpo_trainer.py
Outdated
|
|
||
| from ..data_utils import maybe_apply_chat_template, maybe_extract_prompt | ||
| from .base_trainer import BaseTrainer | ||
| from ..experimental.orpo import ORPOTrainer as ExperimentalORPOTrainer |
Member
There was a problem hiding this comment.
Suggested change
| from ..experimental.orpo import ORPOTrainer as ExperimentalORPOTrainer | |
| from ..experimental.orpo import ORPOTrainer as _ORPOTrainer |
for consistency
trl/trainer/orpo_trainer.py
Outdated
|
|
||
| if is_wandb_available(): | ||
| import wandb | ||
| class ORPOTrainer(ExperimentalORPOTrainer): |
Member
There was a problem hiding this comment.
Suggested change
| class ORPOTrainer(ExperimentalORPOTrainer): | |
| class ORPOTrainer(_ORPOTrainer): |
Member
|
there are some tests in |
qgallouedec
reviewed
Nov 6, 2025
Comment on lines
52
to
54
| from ..base_trainer import BaseTrainer | ||
| from .orpo_config import ORPOConfig | ||
| from ..utils import ( |
Member
There was a problem hiding this comment.
Suggested change
| from ..base_trainer import BaseTrainer | |
| from .orpo_config import ORPOConfig | |
| from ..utils import ( | |
| from ...trainer.base_trainer import BaseTrainer | |
| from .orpo_config import ORPOConfig | |
| from ...trainer.utils import ( |
qgallouedec
reviewed
Nov 6, 2025
|
|
||
| import numpy as np | ||
| import pandas as pd | ||
| import torch |
Member
There was a problem hiding this comment.
torch import should remain, some type hints need it
- Restore ORPO imports in trl/trainer/__init__.py for backward compatibility - Fix deprecation stub naming from ExperimentalORPOTrainer to _ORPOTrainer - Add torch import to deprecation stub for type hints - Fix relative import paths in trl/experimental/orpo/orpo_trainer.py - Update autodoc references to experimental.orpo.ORPOTrainer - Update all documentation references to use experimental namespace - Move ORPO test from test_trainers_args.py to experimental/test_trainers_args.py
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
- Remove unused 'import os' and 'import warnings' from trl/experimental/orpo/orpo_trainer.py - Remove unused 'from typing import Any' from trl/trainer/orpo_trainer.py
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR moves
ORPOTrainerandORPOConfigfromtrl.trainertotrl.experimental.orpoas part of the TRL V1 refactoring effort.Changes
ORPOTrainerandORPOConfigtotrl.experimental.orpotrl.trainerwithFutureWarning(removal planned for TRL 0.29.0)tests/test_orpo_trainer.py)examples/scripts/orpo.py)docs/source/orpo_trainer.md,docs/source/_toctree.yml)trl.trainerexportsTesting
Fixes #4465
Contributes to #4374