[from_pretrained] Fix failing PEFT tests

[younesbelkada]

What does this PR do?

Addresses: #25726 (comment)
In fact it is important to overwrite pretrained_model_name_or_path with the base_model_name_or_path attribute of the adapter config otherwise from_pretrained will try to load the config file from the adapter model_id where it should look for it at base_model_name_or_path

now all PEFT tests are green

cc @sgugger @ArthurZucker

[sgugger]

Thanks for fixing!

[ArthurZucker]

Ouch I think it's a typo from merging see here

[ArthurZucker]

Let's make them not slow otherwise we are shooting in the dark 😉

[younesbelkada]

I agree it sounds important to have it, happy to add a PEFT circleCI job that runs tests_torch as well as PEFT tests, I can do that in a follow up PR, what do you think @sgugger @ydshieh (offline we have agreed to not do it, maybe we should after these failures?)

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ydshieh]

We can have a new job to see how long it takes.

I would try to keep the new job to run only the PEFT tests but not to re-run the tests_torch.

Let's hear our @sgugger 's wisdom too. We will miss it.

[younesbelkada]

Merging as this is blocking for an upcoming PR #25738 , will address future comments in a follow up PR!