eval_loss not found when training a peft model using trainer.py

[ChintanShahDS]

System Info

transformers version: 4.43.3
Python 3.10.12
Ubuntu

Issue is with the trainer.py since it does not check the base_model for Peft cases to get the label information

Have fixed in local but raising this to fix in the branch

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run a vision transformer training using Lora

Expected behavior

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/optimum/habana/transformers/trainer.py", line 1341, in _save_checkpoint
metric_value = metrics[metric_to_check]
KeyError: 'eval_loss'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/vision/run_image_classification.py", line 510, in
main()
File "/home/vision/run_image_classification.py", line 480, in main
train_result = trainer.train(resume_from_checkpoint=checkpoint)
File "/usr/local/lib/python3.10/dist-packages/optimum/habana/transformers/trainer.py", line 553, in train
return inner_training_loop(
File "/usr/local/lib/python3.10/dist-packages/optimum/habana/transformers/trainer.py", line 1052, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, _grad_norm, model, trial, epoch, ignore_keys_for_eval)
File "/usr/local/lib/python3.10/dist-packages/optimum/habana/transformers/trainer.py", line 1269, in _maybe_log_save_evaluate
self._save_checkpoint(model, trial, metrics=metrics)
File "/usr/local/lib/python3.10/dist-packages/optimum/habana/transformers/trainer.py", line 1343, in _save_checkpoint
raise KeyError(
KeyError: "The metric_for_best_model training argument is set to 'eval_loss', which is not found in the evaluation metrics. The available evaluation metrics are: ['eval_runtime', 'eval_samples_per_second', 'eval_steps_per_second', 'epoch', 'memory_allocated (GB)', 'max_memory_allocated (GB)', 'total_memory_available (GB)']. Consider changing the metric_for_best_model via the TrainingArguments."

[ChintanShahDS]

Able to find the problem and the solution.

    # Modified for peft related changes that were missing in the original version
    if _is_peft_model(self.model):
        if hasattr(self.model, "get_base_model"):
            model_to_inspect = self.model.get_base_model()
            default_label_names = find_labels(model_to_inspect.__class__)
            self.can_return_loss = can_return_loss(model_to_inspect.__class__)
    else:
        default_label_names = find_labels(self.model.__class__)
        self.can_return_loss = can_return_loss(self.model.__class__)

    self.label_names = default_label_names if self.args.label_names is None else self.args.label_names
    self.control = self.callback_handler.on_init_end(self.args, self.state, self.control)
    # End of modification

Need to convert to PR so that can pull and make these changes

[ChintanShahDS]

The issue is fixed using the code that is updated on trainer and sent as part of pull request

[JulioSanchezD]

8 months later and I'm still facing this issue with the base library. This solution has proven to work for me by modifying the trainer.py script, now I can monitor the evaluation loss during training. This solution should be merged.

[Rocketknight1]

the PR #33420 was closed soon after it was created so we never reviewed it - @ChintanShahDS does this issue still exist, and would you like to reopen your PR if so?

[ChintanShahDS]

Yes it exists but was not able to get the solution integrated. Though have suggested the solution and used it on my local

…

On Tue, 13 May, 2025, 15:53 Matt, ***@***.***> wrote: *Rocketknight1* left a comment (huggingface/transformers#33420) <#33420 (comment)> the PR #33420 <#33420> was closed soon after it was created so we never reviewed it - @ChintanShahDS <https://github.com/ChintanShahDS> does this issue still exist, and would you like to reopen your PR if so? — Reply to this email directly, view it on GitHub <#33420 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWQOXBEAOD72I3C3YWMRMAD26HBY3AVCNFSM6AAAAABOADEARGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNZVHEZDENJSGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Rocketknight1]

We might be interested in reopening #33422 in that case! If you or anyone else (@JulioSanchezD?) wants to test and open a new PR with those fixes, we can review it

[kreil]

I can confirm the issues persists.

There are further issues with peft / trainer integration that seem to have the same or at least related causes. Essentially, there are other places in trainer.py that also do not go to the base model as required to look up the correct losses, such as the code that loads the best model.

Would you want a separate issue raised, linking to this issue or should this be treated as one issue as the cause is the same?

Essentially, training a peft model with trainer.py and setting to load_best_model_at_end to True throws the below error

File ".../transformers/trainer.py", line 3177, in _determine_best_metric
    raise KeyError(
KeyError: "The `metric_for_best_model` training argument is set to 'eval_loss', which is not found in the evaluation metrics. The available evaluation metrics are: ['eval_runtime', 'eval_samples_per_second', 'eval_steps_per_second', 'epoch']. Consider changing the `metric_for_best_model` via the TrainingArguments."

You can see that trainer lists the peft losses rather than those of the base model as expected / required.

Where should that best be fixed?

Best regards,
David