perf: Avoid copying inputs_embeds tensors to GPU unless prompt_embeds is enabled

[qthequartermasterman]

Purpose

@njhill pointed out that #24278 introduced a performance regression in the case when prompt embeds is disabled, where inputs_embeds tensors are copied to GPU, even though those tensors are either empty or filled with garbage when --enable-prompt-embeds is not on. We can guard those copies with self.enable_prompt_embeds to avoid these copies unless prompt embeds is enabled.

Test Plan

No new tests are needed. I have some local scripts to send thousands of requests through vLLM with this change. Everything works.

Test Result

Relevant local tests pass. Pending CI.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces a performance optimization by avoiding unnecessary GPU copies of inputs_embeds and is_token_ids when prompt embeddings are not enabled. The changes are implemented correctly by adding if self.enable_prompt_embeds guards. I've also pointed out a related pre-existing issue in the async scheduling path where is_token_ids is not fully updated, which would be good to address for complete correctness, though it might be out of scope for this specific performance-focused PR.

[gemini-code-assist]

This change correctly handles is_token_ids for the fast path in async scheduling. However, a similar update is missing for the slow path (the scatter_ case for input_ids around line 900). For correctness when prompt embeddings are enabled, is_token_ids for the scattered tokens should also be set to True, as they correspond to sampled token IDs. A complete fix would handle both paths.

[DarkLight1337]

We can also disable this code?

        is_token_ids = self.input_batch.is_token_ids.flatten()
        torch.index_select(
            is_token_ids,
            0,
            token_indices_tensor,
            out=self.is_token_ids.cpu[:total_num_scheduled_tokens])