[Misc]: benchmark_serving with image input

[Mrxiangli]

Anything you want to discuss about vllm.

Can the current benchmark_serving.py be used with Multimodal LLM (llava) and image input? The existing code send the request in the following format in backend_request_func.py, Is it possible to make it support image input?

        payload = {
            "model": request_func_input.model,
            "messages": [
                {
                    "role": "user",
                    "content": request_func_input.prompt,
                },
            ],
            "temperature": 0.0,
            "max_tokens": request_func_input.output_len,
            "stream": True,
        }
        headers = {
            "Content-Type": "application/json",
            "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}",
        }

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Mrxiangli]

@DarkLight1337 Hello, I saw that you are working on incorporating multimodal into the offline chat. I'm trying to send image input through the benchmark_serving.py while having an openai_api server running in the backend, is this something already being supported? If not? Could you give some suggestions on how to add these features? Thanks!

[DarkLight1337]

You just have to follow the standard OpenAI API (in particular the Vision part in Chat Completions API) when sending the request.

[ywang96]

Hey @Mrxiangli! Thanks for your interest in adding a benchmark for VLMs!

You'll have to modify backend_request_func.py to add a request generator specifically for OpenAI Vision API for VLMs (which shouldn't be a big change from the existing OpenAI API generator other than the payload) since currently the benchmark assumes text-only input. Feel free to ping me whenever you have something ready!