Enable headless models for pooling in the Transformers backend

[hmellor]

Previously, embedding model checkpoints that had their layers at the root of the checkpoint would not load correctly with the Transformers backend.

This PR enables the loading of Transformers base model classes.

Now, both of the following formats of checkpoint will work for pooling tasks:

ModelForCausalLM:

- model
  - layers
    - ...
  - lm_head

Model:

- layers
  - ...
- ...

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request enables loading headless models for embedding with the Transformers backend, which is a great addition. The changes in the configuration and registry look correct, and the new test case covers the intended scenarios.

However, I've found a critical issue in the WeightsMapper implementation for the new TransformersModel. The current logic for prefixing weights is flawed due to incorrect key ordering in the dictionary, which will cause weight loading to fail for one of the model formats this PR aims to support. I've provided a detailed comment and a code suggestion to fix this.

[DarkLight1337]

Thanks for extending this support!

[hmellor]

The mapper is having issues, I'll disable auto-merge for now