how to use V1 engine in vllm mode

[ZCSlqc]

I used A10 GPU to run CosyVoice2-0.5B in VLLM mode, but encountered the message: "--enable-prompt-embeds is not supported by the V1 Engine. Falling back to V0". When I commented out this line in the  load_vllm()  function of  cosyvoice/cli/model.py , the model initialization failed this time.

I would appreciate your help with this issue.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[anxiang1836]

change vllm input from prompt-embeds to prompt_token_ids，reference this repo:
https://github.com/qi-hua/async_cosyvoice

[ZCSlqc]

change vllm input from prompt-embeds to prompt_token_ids，reference this repo: https://github.com/qi-hua/async_cosyvoice

thanks，I will try it

[anxiang1836]

I have another idea.
I have tried to merge the speech tokens to cosyvoice tokenizer (add tokens like <|st_0|>/<|st_1|>....). And change the llm ckpt 'embed_tokens'/'lm_head' weight size to the tokenizer same size, by init llm_head weight 0 (the non-speech token). This will change the ckpt files to the hugging face format and vllm can handle it as a standard Qwen2ForCausalLM.
Note: The vllm output speech token need to shift the tokenizer offset to 0~6561.

[anxiang1836]

vllm(0.11.0) has supported 'enable-prompt-embeds' in v1 engine.
But I caught an exception when start engine, without change any code in this repo. This is the vllm new feature PR link:
vllm-project/vllm#24278