[Bugfix]: make most of `test_openai_schema.py` pass #17664

davidxia · 2025-05-05T15:32:02Z

This change makes most of the parameterized tests in pytest tests/entrypoints/openai/test_openai_schema.py pass. We handle user input errors, enrich HTTP error messages with more information, and update the error response definitions for several API endpoints.

The remaining tests that fail are POST /tokenize and maybe POST /v1/chat/completions.

FIX #17037
FIX #17038

test results before

$ pytest tests/entrypoints/openai/test_openai_schema.py

...

===================================================================================== short test summary info =====================================================================================
(verbose_name='POST /tokenize') SUBFAIL tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /tokenize] - schemathesis.exceptions.CheckFailed: 
(verbose_name='POST /detokenize') SUBFAIL tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /detokenize] - schemathesis.exceptions.CheckFailed: 
(verbose_name='POST /v1/chat/completions') SUBFAIL tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /v1/chat/completions] - schemathesis.exceptions.CheckFailed: 
(verbose_name='POST /v1/completions') SUBFAIL tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /v1/completions] - schemathesis.exceptions.CheckFailed: 
(verbose_name='POST /v1/audio/transcriptions') SUBFAIL tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /v1/audio/transcriptions] - schemathesis.exceptions.CheckFailed: 
(verbose_name='POST /invocations') SUBFAIL tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /invocations] - schemathesis.exceptions.CheckFailed: 
(verbose_name='GET /metrics') SUBFAIL tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless[GET /metrics] - schemathesis.exceptions.CheckFailed: 
============================================================= 7 failed, 1 passed, 9 warnings, 14 subtests passed in 95.93s (0:01:35) ==============================================================

test results after

$ pytest tests/entrypoints/openai/test_openai_schema.py

...

===================================================================================== short test summary info =====================================================================================
(verbose_name='POST /tokenize') SUBFAIL tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /tokenize] - schemathesis.exceptions.CheckFailed: Schemathesis found 2 distinct sets of failures.
(verbose_name='POST /v1/chat/completions') SUBFAIL tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /v1/chat/completions] - schemathesis.exceptions.CheckFailed: Schemathesis found 0 distinct sets of failures.
============================================================= 2 failed, 1 passed, 9 warnings, 19 subtests passed in 267.55s (0:04:27) =============================================================

1. POST /tokenize

before

$ curl -X POST -H 'Content-Type: application/json' -d '{"messages": []}' --insecure http://127.0.0.1:8000/tokenize -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> POST /tokenize HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 16
> 
< HTTP/1.1 400 Bad Request
< date: Mon, 05 May 2025 15:27:25 GMT
< server: uvicorn
< content-length: 80
< content-type: application/json
< 
* Connection #0 to host 127.0.0.1 left intact
{"object":"error","message":"","type":"BadRequestError","param":null,"code":400}

after

$ curl -X POST -H 'Content-Type: application/json' -d '{"messages": []}' --insecure http://127.0.0.1:8000/tokenize -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> POST /tokenize HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 16
> 
< HTTP/1.1 400 Bad Request
< date: Mon, 12 May 2025 16:40:57 GMT
< server: uvicorn
< content-length: 104
< content-type: application/json
< 
* Connection #0 to host 127.0.0.1 left intact
{"object":"error","message":" list index out of range","type":"BadRequestError","param":null,"code":400}

2. POST /detokenize

before

$ curl -X POST -H 'Content-Type: application/json' -d '{"tokens": [-1]}' --insecure http://127.0.0.1:8000/detokenize -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> POST /detokenize HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 16
> 
< HTTP/1.1 500 Internal Server Error
< date: Mon, 05 May 2025 15:27:50 GMT
< server: uvicorn
< content-length: 21
< content-type: text/plain; charset=utf-8
< 
* Connection #0 to host 127.0.0.1 left intact
Internal Server Error

after

$ curl -X POST -H 'Content-Type: application/json' -d '{"tokens": [-1]}' --insecure http://127.0.0.1:8000/detokenize -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> POST /detokenize HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 16
> 
< HTTP/1.1 400 Bad Request
< date: Mon, 12 May 2025 16:41:46 GMT
< server: uvicorn
< content-length: 128
< content-type: application/json
< 
* Connection #0 to host 127.0.0.1 left intact
{"object":"error","message":" ['out of range integral type conversion attempted']","type":"Bad Request","param":null,"code":400}

3. POST /v1/chat/completions

before

$ curl -X POST -H 'Content-Type: application/json' -d '{"messages": []}' --insecure http://127.0.0.1:8000/v1/chat/completions -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> POST /v1/chat/completions HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 16
> 
< HTTP/1.1 400 Bad Request
< date: Mon, 05 May 2025 15:28:16 GMT
< server: uvicorn
< content-length: 80
< content-type: application/json
< 
* Connection #0 to host 127.0.0.1 left intact
{"object":"error","message":"","type":"BadRequestError","param":null,"code":400}

after

$ curl -X POST -H 'Content-Type: application/json' -d '{"messages": []}' --insecure http://127.0.0.1:8000/v1/chat/completions -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> POST /v1/chat/completions HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 16
> 
< HTTP/1.1 400 Bad Request
< date: Mon, 12 May 2025 16:42:10 GMT
< server: uvicorn
< content-length: 104
< content-type: application/json
< 
* Connection #0 to host 127.0.0.1 left intact
{"object":"error","message":" list index out of range","type":"BadRequestError","param":null,"code":400}

4. POST /v1/completions

before

$ curl -X POST -H 'Content-Type: application/json' -d '{"prompt": [-1]}' --insecure http://127.0.0.1:8000/v1/completions -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> POST /v1/completions HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 16
> 
< HTTP/1.1 500 Internal Server Error
< date: Mon, 05 May 2025 15:35:48 GMT
< server: uvicorn
< content-length: 21
< content-type: text/plain; charset=utf-8
< 
* Connection #0 to host 127.0.0.1 left intact
Internal Server Error

after

$ curl -X POST -H 'Content-Type: application/json' -d '{"prompt": [-1]}' --insecure http://127.0.0.1:8000/v1/completions -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> POST /v1/completions HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 16
> 
< HTTP/1.1 400 Bad Request
< date: Mon, 12 May 2025 16:43:17 GMT
< server: uvicorn
< content-length: 123
< content-type: application/json
< 
* Connection #0 to host 127.0.0.1 left intact
{"object":"error","message":"out of range integral type conversion attempted","type":"Bad Request","param":null,"code":400}

5. POST /v1/audio/transcriptions

before

$ curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' -d file=b%27%27 --insecure http://127.0.0.1:8000/v1/audio/
transcriptions -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> POST /v1/audio/transcriptions HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 12
> 
< HTTP/1.1 422 Unprocessable Entity
< date: Mon, 12 May 2025 16:48:48 GMT
< server: uvicorn
< content-length: 65
< content-type: application/json
< 
* Connection #0 to host 127.0.0.1 left intact
{"detail":"Expected 'file' to be a file-like object, not 'str'."}

after

$ curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' -d file=b%27%27 --insecure http://127.0.0.1:8000/v1/audio/transcriptions -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> POST /v1/audio/transcriptions HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 12
> 
< HTTP/1.1 422 Unprocessable Entity
< date: Mon, 12 May 2025 16:43:51 GMT
< server: uvicorn
< content-length: 137
< content-type: application/json
< 
* Connection #0 to host 127.0.0.1 left intact
{"object":"error","message":"Expected 'file' to be a file-like object, not 'str'.","type":"Unprocessable Entity","param":null,"code":422}

6. POST /invocations

before

$ curl -X POST --insecure http://localhost:8000/invocations -v
* Host localhost:8000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8000...
* connect to ::1 port 8000 from ::1 port 48744 failed: Connection refused
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000
> POST /invocations HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 415 Unsupported Media Type
< date: Mon, 05 May 2025 21:01:30 GMT
< server: uvicorn
< content-length: 71
< content-type: application/json
< 
* Connection #0 to host localhost left intact
{"detail":"Unsupported Media Type: Only 'application/json' is allowed"}

$ curl -X POST -H 'Content-Type: application/json' --insecure http://127.0.0.1:8000/invocations -v
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> POST /invocations HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> 
< HTTP/1.1 500 Internal Server Error
< date: Mon, 05 May 2025 15:29:13 GMT
< server: uvicorn
< content-length: 21
< content-type: text/plain; charset=utf-8
< 
* Connection #0 to host 127.0.0.1 left intact
Internal Server Error

after

$ curl -X POST --insecure http://localhost:8000/invocations -v
* Host localhost:8000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8000...
* connect to ::1 port 8000 from ::1 port 35996 failed: Connection refused
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000
> POST /invocations HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 400 Bad Request
< date: Mon, 12 May 2025 16:44:29 GMT
< server: uvicorn
< content-length: 141
< content-type: application/json
< 
* Connection #0 to host localhost left intact
{"object":"error","message":" [\"Unsupported Media Type: Only 'application/json' is allowed\"]","type":"Bad Request","param":null,"code":400}

$ curl -X POST -H 'Content-Type: application/json' --insecure http://127.0.0.1:8000/invocations -v
*   Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> POST /invocations HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> 
< HTTP/1.1 400 Bad Request
< date: Mon, 12 May 2025 16:44:09 GMT
< server: uvicorn
< content-length: 136
< content-type: application/json
< 
* Connection #0 to host 127.0.0.1 left intact
{"object":"error","message":"JSON decode error: Expecting value: line 1 column 1 (char 0)","type":"Bad Request","param":null,"code":400}

github-actions · 2025-05-05T15:32:12Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

vllm/entrypoints/openai/api_server.py

tests/entrypoints/openai/test_openai_schema.py

vllm/entrypoints/openai/api_server.py

davidxia · 2025-05-05T20:08:31Z

vllm/entrypoints/openai/api_server.py

@router.api_route isn't recommended. See fastapi/fastapi#7736 (comment)

response_class=Response fixes errors

___________________________________________________ test_openapi_stateless (verbose_name='POST /ping') ____________________________________________________ @wraps(test) > def test_function(*args: Any, **kwargs: Any) -> Any: .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:83: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:160: in async_run loop.run_until_complete(future) /usr/lib/python3.12/asyncio/base_events.py:687: in run_until_complete return future.result() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ case = Case() @schema.parametrize() @schema.override(headers={"Content-Type": "application/json"}) async def test_openapi_stateless(case): #No need to verify SSL certificate for localhost > await case.call_and_validate(verify=False) E schemathesis.exceptions.CheckFailed: E E 1. Missing Content-Type header E E The following media types are documented in the schema: E - `application/json` E E [200] OK: E E <EMPTY> E E Reproduce with: E E curl -X POST --insecure http://localhost:52711/ping E E Falsifying example: async_run( E case=, E ) tests/entrypoints/openai/test_openai_schema.py:49: CheckFailed

davidxia · 2025-05-05T20:09:09Z

vllm/entrypoints/openai/api_server.py

fixes errors like

___________________________________________________ test_openapi_stateless (verbose_name='GET /health') ___________________________________________________ @wraps(test) > def test_function(*args: Any, **kwargs: Any) -> Any: .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:83: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:160: in async_run loop.run_until_complete(future) /usr/lib/python3.12/asyncio/base_events.py:687: in run_until_complete return future.result() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ case = Case() @schema.parametrize() @schema.override(headers={"Content-Type": "application/json"}) async def test_openapi_stateless(case): #No need to verify SSL certificate for localhost > await case.call_and_validate(verify=False) E schemathesis.exceptions.CheckFailed: E E 1. Missing Content-Type header E E The following media types are documented in the schema: E - `application/json` E E [200] OK: E E <EMPTY> E E Reproduce with: E E curl -X GET --insecure http://localhost:52711/health E E Falsifying example: async_run( E case=, E ) tests/entrypoints/openai/test_openai_schema.py:49: CheckFailed

.buildkite/test-pipeline.yaml

davidxia · 2025-05-06T00:36:55Z

vllm/entrypoints/openai/api_server.py

We raise a RequestValidationError so that the existing validation_exception_handler() will convert it to a JSONResponse. We need that to happen so we can specify ErrorResponse as the return type in API endpoints like @router.post("/invocations", ...). FastAPI requires return types to be Pydantic types. ErrorResponse inherits from Pydantic's BaseModel.

Otherwise fails with errors like

__________________________________________________________________________ test_openapi_stateless (verbose_name='POST /v1/completions') ___________________________________________________________________________ @wraps(test) > def test_function(*args: Any, **kwargs: Any) -> Any: .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:83: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:160: in async_run loop.run_until_complete(future) /usr/lib/python3.12/asyncio/base_events.py:687: in run_until_complete return future.result() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ case = Case(body={'prompt': [], 'logprobs': -1}) @schema.parametrize() @schema.override(headers={"Content-Type": "application/json"}) async def test_openapi_stateless(case): # No need to verify SSL certificate for localhost > case.call_and_validate(verify=False) E schemathesis.exceptions.CheckFailed: E E 1. Server error E E 2. Undocumented Content-Type E E Received: text/plain; charset=utf-8 E Documented: application/json E E [500] Internal Server Error: E E `Internal Server Error` E E Reproduce with: E E curl -X POST -H 'Content-Type: application/json' -d '{"prompt": [], "logprobs": -1}' --insecure http://localhost:45731/v1/completions E E Falsifying example: async_run( E case=, E ) tests/entrypoints/openai/test_openai_schema.py:49: CheckFailed ____________________________________________________________________________ test_openapi_stateless (verbose_name='POST /invocations') ____________________________________________________________________________ @wraps(test) > def test_function(*args: Any, **kwargs: Any) -> Any: .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:83: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:160: in async_run loop.run_until_complete(future) /usr/lib/python3.12/asyncio/base_events.py:687: in run_until_complete return future.result() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ case = Case() @schema.parametrize() @schema.override(headers={"Content-Type": "application/json"}) async def test_openapi_stateless(case): # No need to verify SSL certificate for localhost > case.call_and_validate(verify=False) E schemathesis.exceptions.CheckFailed: E E 1. Response violates schema E E 'message' is a required property E E Schema: E E { E "properties": { E "object": { E "type": "string", E "title": "Object", E "default": "error" E }, E "message": { E "type": "string", E "title": "Message" E }, E "type": { E "type": "string", E "title": "Type" E }, E "param": { E "anyOf": [ E { E "type": "string" E // Output truncated... E } E E Value: E E { E "detail": "Unsupported Media Type: Only 'application/json' is allowed" E } E E [415] Unsupported Media Type: E E `{"detail":"Unsupported Media Type: Only 'application/json' is allowed"}` E E Reproduce with: E E curl -X POST --insecure http://localhost:45731/invocations E E Falsifying example: async_run( E case=, E ) tests/entrypoints/openai/test_openai_schema.py:49: CheckFailed

vllm/entrypoints/openai/api_server.py

tests/entrypoints/openai/test_openai_schema.py

vllm/entrypoints/openai/api_server.py

davidxia · 2025-05-12T13:47:37Z

vllm/entrypoints/openai/api_server.py

fixes errors like

__________________________________________ test_openapi_stateless (verbose_name='POST /v1/audio/transcriptions') __________________________________________ @wraps(test) > def test_function(*args: Any, **kwargs: Any) -> Any: .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:83: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:160: in async_run loop.run_until_complete(future) /usr/lib/python3.12/asyncio/base_events.py:687: in run_until_complete return future.result() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ case = Case(body={'file': Binary(data=b'')}) @schema.parametrize() @schema.override(headers={"Content-Type": "application/json"}) async def test_openapi_stateless(case): #No need to verify SSL certificate for localhost > await case.call_and_validate(verify=False) E schemathesis.exceptions.CheckFailed: E E 1. Undocumented HTTP status code E E Received: 400 E Documented: 200, 422 E E [400] Bad Request: E E `{"object":"error","message":"[{'type': 'value_error', 'loc': ('body', 'file'), 'msg': \"Value error, Expected UploadFile, received: <class 'str'>\", 'input': \"b''\", 'ctx': {'error': ValueError(\"Expected UploadFile, received: <class 'str'>\")}}]","type":"BadRequestError","param":null,"code":400}` E E Reproduce with: E E curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' -d file=b%27%27 --insecure http://localhost:52711/v1/audio/transcriptions E E Falsifying example: async_run( E case=, E ) tests/entrypoints/openai/test_openai_schema.py:49: CheckFailed

davidxia · 2025-05-12T13:48:28Z

vllm/entrypoints/openai/api_server.py

fixes errors like

______________________________________________ test_openapi_stateless (verbose_name='POST /v1/completions') _______________________________________________ @wraps(test) > def test_function(*args: Any, **kwargs: Any) -> Any: .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:83: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:160: in async_run loop.run_until_complete(future) /usr/lib/python3.12/asyncio/base_events.py:687: in run_until_complete return future.result() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ case = Case(body={'prompt': []}) @schema.parametrize() @schema.override(headers={"Content-Type": "application/json"}) async def test_openapi_stateless(case): #No need to verify SSL certificate for localhost > await case.call_and_validate(verify=False) E schemathesis.exceptions.CheckFailed: E E 1. Undocumented HTTP status code E E Received: 400 E Documented: 200, 422 E E [400] Bad Request: E E `{"object":"error","message":"please provide at least one prompt","type":"BadRequestError","param":null,"code":400}` E E Reproduce with: E E curl -X POST -H 'Content-Type: application/json' -d '{"prompt": []}' --insecure http://localhost:52711/v1/completions E E Falsifying example: async_run( E case=, E ) tests/entrypoints/openai/test_openai_schema.py:49: CheckFailed

davidxia · 2025-05-12T13:48:53Z

vllm/entrypoints/openai/api_server.py

fixes errors like

____________________________________________ test_openapi_stateless (verbose_name='POST /v1/chat/completions') ____________________________________________ @wraps(test) > def test_function(*args: Any, **kwargs: Any) -> Any: .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:83: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:160: in async_run loop.run_until_complete(future) /usr/lib/python3.12/asyncio/base_events.py:687: in run_until_complete return future.result() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ case = Case(body={'messages': []}) @schema.parametrize() @schema.override(headers={"Content-Type": "application/json"}) async def test_openapi_stateless(case): #No need to verify SSL certificate for localhost > await case.call_and_validate(verify=False) E schemathesis.exceptions.CheckFailed: E E 1. Undocumented HTTP status code E E Received: 400 E Documented: 200, 422 E E [400] Bad Request: E E `{"object":"error","message":"","type":"BadRequestError","param":null,"code":400}` E E Reproduce with: E E curl -X POST -H 'Content-Type: application/json' -d '{"messages": []}' --insecure http://localhost:52711/v1/chat/completions E E Falsifying example: async_run( E case=, E ) tests/entrypoints/openai/test_openai_schema.py:49: CheckFailed

davidxia · 2025-05-12T13:49:40Z

vllm/entrypoints/openai/api_server.py

fixes errors like

_________________________________________________ test_openapi_stateless (verbose_name='POST /tokenize') __________________________________________________ @wraps(test) > def test_function(*args: Any, **kwargs: Any) -> Any: .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:83: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:160: in async_run loop.run_until_complete(future) /usr/lib/python3.12/asyncio/base_events.py:687: in run_until_complete return future.result() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ case = Case(body={'messages': []}) @schema.parametrize() @schema.override(headers={"Content-Type": "application/json"}) async def test_openapi_stateless(case): #No need to verify SSL certificate for localhost > await case.call_and_validate(verify=False) E schemathesis.exceptions.CheckFailed: E E 1. Undocumented HTTP status code E E Received: 400 E Documented: 200, 422 E E [400] Bad Request: E E `{"object":"error","message":"","type":"BadRequestError","param":null,"code":400}` E E Reproduce with: E E curl -X POST -H 'Content-Type: application/json' -d '{"messages": []}' --insecure http://localhost:52711/tokenize E E Falsifying example: async_run( E case=, E ) tests/entrypoints/openai/test_openai_schema.py:49: CheckFailed

davidxia · 2025-05-12T13:52:27Z

vllm/entrypoints/openai/api_server.py

Needed to test expecting Content-Type: application/json all the time. See https://fastapi.tiangolo.com/advanced/additional-responses/#additional-media-types-for-the-main-response

davidxia · 2025-05-12T13:53:57Z

vllm/entrypoints/openai/api_server.py

changed to return a Response since there's no JSON payload to return

davidxia · 2025-05-12T13:56:49Z

This looks nice so far, although I'm not sure it solves #17038 as that issue says that some endpoints are returning 500 where they should be returning 422.

All the requests that receive unexpected 500 responses documented in #17038 now receive 400. I think this PR now fixes that issue.

``` $ curl -X POST -H 'Content-Type: application/json' -d '{"messages": []}' --insecure http://127.0.0.1:8000/tokenize -v Note: Unnecessary use of -X or --request, POST is already inferred. * Trying 127.0.0.1:8000... * Connected to 127.0.0.1 (127.0.0.1) port 8000 > POST /tokenize HTTP/1.1 > Host: 127.0.0.1:8000 > User-Agent: curl/8.5.0 > Accept: */* > Content-Type: application/json > Content-Length: 16 > < HTTP/1.1 400 Bad Request < date: Mon, 12 May 2025 12:54:20 GMT < server: uvicorn < content-length: 104 < content-type: application/json < * Connection #0 to host 127.0.0.1 left intact {"object":"error","message":" list index out of range","type":"BadRequestError","param":null,"code":400}

$ curl -X POST -H 'Content-Type: application/json' -d '{"tokens": [-1]}' --insecure http://127.0.0.1:8000/detokenize -v
Note: Unnecessary use of -X or --request, POST is already inferred.

Trying 127.0.0.1:8000...
Connected to 127.0.0.1 (127.0.0.1) port 8000

POST /detokenize HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: curl/8.5.0
Accept: /
Content-Type: application/json
Content-Length: 16

< HTTP/1.1 400 Bad Request
< date: Mon, 12 May 2025 13:26:25 GMT
< server: uvicorn
< content-length: 128
< content-type: application/json
<

Connection #0 to host 127.0.0.1 left intact
{"object":"error","message":" ['out of range integral type conversion attempted']","type":"Bad Request","param":null,"code":400}

$ curl -X POST -H 'Content-Type: application/json' -d '{"messages": []}' --insecure http://127.0.0.1:8000/v1/chat/completions -v
Note: Unnecessary use of -X or --request, POST is already inferred.

Trying 127.0.0.1:8000...
Connected to 127.0.0.1 (127.0.0.1) port 8000

POST /v1/chat/completions HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: curl/8.5.0
Accept: /
Content-Type: application/json
Content-Length: 16

< HTTP/1.1 400 Bad Request
< date: Mon, 12 May 2025 13:26:58 GMT
< server: uvicorn
< content-length: 104
< content-type: application/json
<

Connection #0 to host 127.0.0.1 left intact
{"object":"error","message":" list index out of range","type":"BadRequestError","param":null,"code":400}

$ curl -X POST -H 'Content-Type: application/json' -d '{"prompt": [-1]}' --insecure http://127.0.0.1:8000/v1/completions -v
Note: Unnecessary use of -X or --request, POST is already inferred.

Trying 127.0.0.1:8000...
Connected to 127.0.0.1 (127.0.0.1) port 8000

POST /v1/completions HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: curl/8.5.0
Accept: /
Content-Type: application/json
Content-Length: 16

< HTTP/1.1 400 Bad Request
< date: Mon, 12 May 2025 13:30:40 GMT
< server: uvicorn
< content-length: 123
< content-type: application/json
<

Connection #0 to host 127.0.0.1 left intact
{"object":"error","message":"out of range integral type conversion attempted","type":"Bad Request","param":null,"code":400}

$ curl -X POST -H 'Content-Type: application/json' --insecure http://127.0.0.1:8000/invocations -v

Trying 127.0.0.1:8000...
Connected to 127.0.0.1 (127.0.0.1) port 8000

POST /invocations HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: curl/8.5.0
Accept: /
Content-Type: application/json

< HTTP/1.1 400 Bad Request
< date: Mon, 12 May 2025 13:31:35 GMT
< server: uvicorn
< content-length: 136
< content-type: application/json
<

Connection #0 to host 127.0.0.1 left intact
{"object":"error","message":"JSON decode error: Expecting value: line 1 column 1 (char 0)","type":"Bad Request","param":null,"code":400}

</details>

hmellor · 2025-05-12T14:03:05Z

All the requests that receive unexpected 500 responses documented in #17038 now receive 400. I think this PR now fixes that issue.

Shouldn't they recieve 422? That's what the linked issue seems to suggest

davidxia · 2025-05-12T14:04:19Z

This is a good point actually! Could you also reduce reprtition by allowing the full HTTPStatus.XYZ to be passed to create_error_code and then the message, type and code (if not explicitly specified) can be extracted from the HTTPStatus?

I actually just replaced almost all the return ErrorResponse() in the endpoints with raise HTTPException() because the former returned a 200 OK response instead of the expected HTTP error codes. Ready for another review.

davidxia · 2025-05-12T14:07:41Z

All the requests that receive unexpected 500 responses documented in #17038 now receive 400. I think this PR now fixes that issue.

Shouldn't they recieve 422? That's what the linked issue seems to suggest

The error in that issue is saying that only 200 and 422 are documented. I've added more expected codes like 400. HTTP 422 specifically points to semantic or validation errors with the request payload, HTTP 400 broadly indicates issues with the request syntax or structure. Happy to change to whatever makes more sense. lmk!

davidxia · 2025-05-12T14:10:13Z

vllm/entrypoints/openai/api_server.py

This is needed to fix errors like the below where the FastAPI middleware returns an error before our custom endpoint function runs. We need to return a payload that has the schema of ErrorResponse because we document that as the return type in our endpoints.

______________________________________________________________ test_openapi_stateless (verbose_name='POST /v1/audio/transcriptions') ______________________________________________________________ @wraps(test) > def test_function(*args: Any, **kwargs: Any) -> Any: .venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:83: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ case = Case(body={'file': Binary(data=b'')}) @schema.parametrize() @schema.override(headers={"Content-Type": "application/json"}) def test_openapi_stateless(case: schemathesis.Case): key = ( case.operation.method.upper(), case.operation.path, ) timeout = { ("POST", "/v1/chat/completions"): 99999, }.get(key, 10) #No need to verify SSL certificate for localhost > case.call_and_validate(verify=False, timeout=timeout) E schemathesis.exceptions.CheckFailed: E E 1. Response violates schema E E 'message' is a required property E E Schema: E E { E "properties": { E "object": { E "type": "string", E "title": "Object", E "default": "error" E }, E "message": { E "type": "string", E "title": "Message" E }, E "type": { E "type": "string", E "title": "Type" E }, E "param": { E "anyOf": [ E { E "type": "string" E // Output truncated... E } E E Value: E E { E "detail": "Expected 'file' to be a file-like object, not 'str'." E } E E [422] Unprocessable Entity: E E `{"detail":"Expected 'file' to be a file-like object, not 'str'."}` E E Reproduce with: E E curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' -d file=b%27%27 --insecure http://localhost:36871/v1/audio/transcriptions E E Falsifying example: test_openapi_stateless( E case=, E ) tests/entrypoints/openai/test_openai_schema.py:57: CheckFailed

hmellor

This is an exceptionally well documented PR!

I only have one tiny nit about including some of the context you provided on GitHub into code comments

vllm/entrypoints/openai/api_server.py

davidxia

@hmellor thank you. I updated the PR description after rebasing. Ready for another review. 🙏

vllm/entrypoints/openai/api_server.py

davidxia · 2025-05-12T17:40:44Z

The only test that consistently fails for me now is tests/entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /tokenize] caused by

vllm/vllm/entrypoints/chat_utils.py

Line 1083 in 98ea356

raise NotImplementedError(f"Unknown part type: {part_type}")

when part_type is "file".

============================================================================================ FAILURES =============================================================================================
_____________________________________________________________________ test_openapi_stateless (verbose_name='POST /tokenize') ______________________________________________________________________

    @wraps(test)
>   def test_function(*args: Any, **kwargs: Any) -> Any:

.venv/lib/python3.12/site-packages/schemathesis/_hypothesis.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

case = Case(body={'messages': [{'content': [{'file': {}, 'type': 'file'}], 'role': 'user'}]})

    @schema.parametrize(tag="debug")
    @schema.override(headers={"Content-Type": "application/json"})
    def test_openapi_stateless(case: schemathesis.Case):
        key = (
            case.operation.method.upper(),
            case.operation.path,
        )
        timeout = {
            ("POST", "/v1/chat/completions"): 99999,
        }.get(key, 10)
    
        #No need to verify SSL certificate for localhost
>       case.call_and_validate(verify=False, timeout=timeout)
E       schemathesis.exceptions.CheckFailed: 
E       
E       1. Server error
E       
E       [500] Internal Server Error:
E       
E           `{"object":"error","message":"Unknown part type: file","type":"Internal Server Error","param":null,"code":500}`
E       
E       Reproduce with: 
E       
E           curl -X POST -H 'Content-Type: application/json' -d '{"messages": [{"content": [{"file": {}, "type": "file"}], "role": "user"}]}' --insecure http://localhost:48767/tokenize
E       
E       Falsifying example: test_openapi_stateless(
E           case=,
E       )

tests/entrypoints/openai/test_openai_schema.py:57: CheckFailed

$ curl -X POST -H 'Content-Type: application/json' -d '{"messages": [{"content": [{"file": {}, "type": "file"}], "role": "user"}]}' --insecure http://localhost:8000/tokenize -v
Note: Unnecessary use of -X or --request, POST is already inferred.
* Host localhost:8000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8000...
* connect to ::1 port 8000 from ::1 port 40420 failed: Connection refused
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000
> POST /tokenize HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 75
> 
< HTTP/1.1 500 Internal Server Error
< date: Mon, 12 May 2025 17:32:11 GMT
< server: uvicorn
< content-length: 109
< content-type: application/json
< 
* Connection #0 to host localhost left intact
{"object":"error","message":"Unknown part type: file","type":"Internal Server Error","param":null,"code":500}

davidxia · 2025-05-13T12:02:10Z

This is ready for review 🙏

This change makes most of the parameterized tests in `pytest tests/entrypoints/openai/test_openai_schema.py` pass. We handle user input errors, enrich HTTP error messages with more information, and update the error response definitions for several API endpoints. The remaining tests that fail are `POST /tokenize` and maybe `POST /v1/chat/completions`. FIX vllm-project#17037 FIX vllm-project#17038 Signed-off-by: David Xia <[email protected]>

davidxia · 2025-05-14T16:37:22Z

I rebased on main and updated POST /tokenize to return a 501 Not Implemented instead of 500 if a Python NotImplemented exception is raised. It doesn't fix the test because according to the OpenAPI spec, $ curl -X POST -H 'Content-Type: application/json' -d '{"messages": [{"content": [{"file": {}, "type": "file"}], "role": "user"}]}' --insecure http://localhost:8000/tokenize is a valid request that should get a 200 reply. But at least the reply status code is more specific.

We can discuss how to make the test case pass in a follow-up PR and then enabling the overall test by removing the --ignore=entrypoints/openai/test_openai_schema.py from test-pipeline.yaml. But I think this PR is definitely an improvement over the status quo.

davidxia · 2025-05-14T18:05:04Z

I created a follow-up issue #18162.

Signed-off-by: Yuqi Zhang <[email protected]>

mergify bot added the frontend label May 5, 2025

This was referenced May 5, 2025

feat(api_server.py): update error response handling for accurate OpenAPI schema #17626

Closed

[Bug]: Undocumented HTTP Status Codes for vllm endpoints #17037

Closed

davidxia force-pushed the patch5 branch from e9a0aa8 to 0d57552 Compare May 5, 2025 15:42

davidxia commented May 5, 2025

View reviewed changes

vllm/entrypoints/openai/api_server.py Outdated Show resolved Hide resolved

davidxia mentioned this pull request May 5, 2025

[Bug]: Many endpoints are returning 500 Internal Server Error #17038

Closed

1 task

davidxia force-pushed the patch5 branch 3 times, most recently from 5727d39 to ad28637 Compare May 5, 2025 19:56

davidxia changed the title ~~[Bugfix]: handle user input errors, enrich errors~~ [Bugfix]: make test_openai_schema.py pass May 5, 2025

davidxia force-pushed the patch5 branch from ad28637 to 10a43ca Compare May 5, 2025 19:59

mergify bot added the ci/build label May 5, 2025

davidxia commented May 5, 2025

View reviewed changes

davidxia marked this pull request as ready for review May 5, 2025 20:28

davidxia requested review from DarkLight1337, robertgshaw2-redhat and simon-mo as code owners May 5, 2025 20:28

davidxia force-pushed the patch5 branch from 10a43ca to ba578dc Compare May 5, 2025 21:12

This comment was marked as resolved.

Sign in to view

davidxia force-pushed the patch5 branch from ba578dc to 68a0498 Compare May 6, 2025 00:18

davidxia commented May 6, 2025

View reviewed changes

davidxia force-pushed the patch5 branch 3 times, most recently from 613b3f8 to cba53c0 Compare May 6, 2025 02:56

davidxia commented May 6, 2025

View reviewed changes

vllm/entrypoints/openai/api_server.py Outdated Show resolved Hide resolved

vllm/entrypoints/openai/api_server.py Outdated Show resolved Hide resolved

davidxia force-pushed the patch5 branch from cba53c0 to 19bbb69 Compare May 6, 2025 02:59

davidxia commented May 6, 2025

View reviewed changes

tests/entrypoints/openai/test_openai_schema.py Outdated Show resolved Hide resolved

davidxia force-pushed the patch5 branch 2 times, most recently from fbf4491 to 8f1f856 Compare May 9, 2025 13:22

davidxia commented May 12, 2025

View reviewed changes

davidxia requested a review from hmellor May 12, 2025 14:14

hmellor approved these changes May 12, 2025

View reviewed changes

vllm/entrypoints/openai/api_server.py Outdated Show resolved Hide resolved

davidxia force-pushed the patch5 branch from fc2b810 to cff6693 Compare May 12, 2025 16:51

davidxia commented May 12, 2025

View reviewed changes

vllm/entrypoints/openai/api_server.py Outdated Show resolved Hide resolved

davidxia changed the title ~~[Bugfix]: make test_openai_schema.py pass~~ [Bugfix]: make most of test_openai_schema.py pass May 12, 2025

davidxia force-pushed the patch5 branch from cff6693 to cb42a6e Compare May 12, 2025 17:37

davidxia force-pushed the patch5 branch from cb42a6e to 17202ee Compare May 14, 2025 16:34

hmellor enabled auto-merge (squash) May 14, 2025 16:40

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label May 14, 2025

davidxia mentioned this pull request May 14, 2025

[Bug]: make test_openai_schema.py pass, enable it in CI #18162

Closed

1 task

simon-mo merged commit f25e0d1 into vllm-project:main May 15, 2025
67 of 69 checks passed

davidxia deleted the patch5 branch May 15, 2025 00:12

zzzyq pushed a commit to zzzyq/vllm that referenced this pull request May 24, 2025

[Bugfix]: make most of test_openai_schema.py pass (vllm-project#17664)

f35618b

Signed-off-by: Yuqi Zhang <[email protected]>

ApsarasX mentioned this pull request Oct 13, 2025

[Frontend] Refactor error handling for multiple exceptions in preprocessing #26694

Open

5 tasks

Uh oh!

[Bugfix]: make most of test_openai_schema.py pass #17664

[Bugfix]: make most of test_openai_schema.py pass #17664

Uh oh!

Conversation

davidxia commented May 5, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. POST /tokenize

2. POST /detokenize

3. POST /v1/chat/completions

4. POST /v1/completions

5. POST /v1/audio/transcriptions

6. POST /invocations

Uh oh!

github-actions bot commented May 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davidxia May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidxia May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as resolved.

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidxia commented May 12, 2025

Uh oh!

hmellor commented May 12, 2025

Uh oh!

davidxia commented May 12, 2025

Uh oh!

davidxia commented May 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hmellor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davidxia left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davidxia commented May 12, 2025

Uh oh!

davidxia commented May 13, 2025

Uh oh!

davidxia commented May 14, 2025

Uh oh!

davidxia commented May 14, 2025

Uh oh!

Uh oh!

Reviewers

[Bugfix]: make most of `test_openai_schema.py` pass #17664

[Bugfix]: make most of `test_openai_schema.py` pass #17664

davidxia commented May 5, 2025 •

edited by github-actions bot

Loading

davidxia May 5, 2025 •

edited

Loading

davidxia May 5, 2025 •

edited

Loading

davidxia left a comment •

edited

Loading