eval_loss not found when training a peft model using trainer.py / losses not retrieved from base model where appropriate

[kreil]

System Info

pip list |grep transf; python --version; uname -a

transformers 4.51.3
Python 3.12.10
Linux gap 6.11.10+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.11.10-1~bpo12+1 (2024-12-19) x86_64 GNU/Linux

Who can help?

@zach-huggingface @SunMarc

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Train any peft model. I am training a LoRA adapter using the AutoModelForMaskedLM from ModernBERT.

Expected behavior

Trainer and its most regular features should handle looking up the losses in the base model, which at several places in the code they apparently don't. So instead of finding eval_loss they only see the new losses added by peft such as eval_steps_per_second.

One such issue is described, including a fix, in #33420. This fix should be merged with the regular sources.

The second such issue, triggered by setting load_best_model_at_end to True, is described in the comments to #33420 (no fix yet).

[Rocketknight1]

cc @BenjaminBossan and @SunMarc

[SunMarc]

Hey @kreil, if you are share a reproducer, that would help us fix the bug. Happyy to review the PR if you have the solution also !

[artintel]

I also had this problem and solved it according to #33420 hopefully this pr will be merge

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[HuChundong]

why this pr not merged? i meet this issue in transformers==4.52.4

[Jacky-MYQ]

"The second such issue, triggered by setting load_best_model_at_end to True, is described in the comments to #33420 (no fix yet)."
yep! i think so ! when i use "load_best_model_at_end", i will have the same problem. Has anyone solved this problem?

[Jacky-MYQ]

The second such issue, triggered by setting load_best_model_at_end to True, is described in the comments to #33420 (no fix yet).

emmm, i saw the code, i found that we need to set label_names this args (by the way it is a List[str]) in TrainingArguments, and it will not emerge "The metric_for_best_model training argument is set to 'eval_loss', which is not found in the evaluation metrics. The available evaluation metrics are: ['eval_runtime', 'eval_samples_per_second', 'eval_steps_per_second', 'epoch']. Consider changing the metric_for_best_model via the TrainingArguments." again when we set load_best_model_at_end to True.