[Doc]: Newest documentation for engine arguments is significantly worse than v0.8.5 and prior

[gjsmo]

📚 The doc issue

The current documentation directs users to the developer documentation of EngineArgs, AsyncEngineArgs, vllm.config, and AsyncLLMEngine to see available engine arguments. This is considerably less useful than the previous version of the page (v0.8.5.post1 and prior, I believe), for a few reasons:

• There are now four potential pages to look at, and it's not immediately clear which one the user needs to go to for the information they need. It seems to be vllm.config for most things, but not all?
• The various pages are developer documentation, and reading them requires wading through all the various methods and additional jargon regarding implementation details.
• These pages do not seem to provide a list of the possible values for many options, for instance ModelConfig.quantization. It is of course possible to click on QuantizationMethods to get the appropriate information, but again this was much easier to view in the previous version of the page.

Suggest a potential alternative/fix

The User Guide should not link to developer documentation, IMO. While it's not entirely unreasonable to expect users of vLLM to be able to read and understand this (as indeed I can understand the new documentation), it's far more work than should be necessary from a user who isn't also a developer. The User Guide should have a comprehensive list of the options available with vllm serve rather than links to developer documentation or telling people to run vllm serve --help (which isn't great for search and discoverability). Notably, the output of vllm serve --help is actually quite good, and does contain everything a user should need - perhaps a documentation page can be autogenerated from this?

Minor edit: it looks like #18264 actually did this, but then by the time #18661 was merged it was in its current form, and in the middle something odd happened with #18145 causing the git history to be a bit weird as it seems to think the file was renamed from something else. It looks like the content as of #18264 would be decent.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[tremble]

The User Guide should have a comprehensive list of the options available with vllm serve rather than links to developer documentation or telling people to run vllm serve --help (which isn't great for search and discoverability). Notably, the output of vllm serve --help is actually quite good, and does contain everything a user should need - perhaps a documentation page can be autogenerated from this?

As someone who uses vLLM in a container inside OpenShift, using vllm serve --help is potentially more annoying to access than for developers (Not impossible, just extra steps). As such something as simple as an auto-generated page would be very helpful.

[DarkLight1337]

Can you check out the docs in #18666 and see if it's better? I've updated the index page of Configuration section so now it actually points to the pages from User Guide.

[hmellor]

perhaps a documentation page can be autogenerated from this?

It was auto-generated before. However, this was problematic because it meant having to be able to execute vLLM in the docs build environment.

the output of vllm serve --help is actually quite good

Then output of this command actually comes from the docstrings, type hints and defaults in vllm.config, but I conceed that it's not as simply presented in the API reference as in the help command

[hmellor]

We could bring this part of the documentation back using a pre-commit hook which updates some autogenerated files when config.py (or the file that creates the CLI) is changed. We could make it run on the developer's system, rather than in an isolated environment, so pre-commit doesn't need its own vLLM installation.

[gjsmo]

@DarkLight1337 I think that's an improvement but it's not really my primary issue, as it is certainly possible to access the various pages. The pain point is really the content not providing the full, verbose list of CLI args.

@hmellor I see, it definitely seems like it would be annoying to have CI have to actually load a full instance of vLLM just to generate docs. I'm not familiar with MkDocs but I do wonder if it would be possible to essentially parse config.py and generate help in the same way it would be done when run from command line. Might be more effort than it's worth vs. a commit hook.

[hmellor]

It doesn't need a full vLLM instance, just to import https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py to access the EngineArgs.add_cli_args method.

The problem is that this requires an enviroment where this file is importable (and all files it imports too), which either leads to complex dependency mocking (maintenance burden), or actually installing vLLM in your docs environment (build time burden).

[strangiato]

I wanted to jump in and add some additional support for bringing back the detailed output similar to vllm serve --help.

I probably reference the engine args page 2-3 times a week personally and it is very helpful when troubleshooting issues between versions. The arguments and options are changing so rapidly that it is VERY important to have a static reference per version that you can access outside of having to install another version of vLLM just to check an argument.

We could bring this part of the documentation back using a pre-commit hook

I would argue this is an anti-pattern.

Pre-commit hooks are great for reducing the churn and helping users to understand what standards they should be following, but anything that is a pre-commit hook should be enforced via CI.

The docs will eventually end up out of sync because someone did a PR without running the hooks, and the next time someone else does a PR and they do run the hooks they are going to end up with un-expected changes.

I'm not sure of all of the details of what you are attempting to optimize for or what challenges the previous CI was causing, but I do think that relying on pre-commit hooks for this is going to cause more pain than just bringing back the old CI that was generating these.

[hmellor]

it is VERY important to have a static reference per version that you can access outside of having to install another version of vLLM just to check an argument.

We still have this. All the information from vllm serve --help comes from https://docs.vllm.ai/en/latest/api/vllm/config.html

but anything that is a pre-commit hook should be enforced via CI.

Our pre-commit hooks are enforced via CI

bringing back the old CI that was generating these.

There was no separate CI generating these docs, it was a Sphinx extension. Since we no longer use Sphinx, we cannot bring back the old solution

[hmellor]

Having said that, the same code that's used to generate the help docs could be run as a pre-commit hook or a mkdocs hook, so it could be part of the docs build again. We'd just have to (as we did before) create and maintain a minimal environment that allows the parsers to be executed.

[MikeSpreitzer]

Getting the command line arguments documentation should be easy!
I should not have to set up an environment.
I tried docker run vllm/vllm-openai vllm serve --help.
I spent more time than I should have on a machine without enough free disk space to even download and unpack that huge image.
When I finally got past that, it crashed like this:

...
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 2245, in __post_init__
    raise RuntimeError(
RuntimeError: Failed to infer device type, please set the environment variable `VLLM_LOGGING_LEVEL=DEBUG` to turn on verbose logging to help debug the issue.

[hmellor]

Using the Docker image is a great idea for this! I won't be able to work on this soon, but I can try this approach once I'm back from vacation.

My rough plan would be:

• Write an argparse formatter which outputs the --help command in markdown
• Use the nightly Docker image from AWS ECR to run vllm serve --help
• Create a MkDocs hook which runs this only if Docker is installed on the docs build machine
• Write the output to a file which is .gitignoreed and then included with --<-- path/to/file.md so that users building the docs do not need to commit the output

[Gongzq5]

Hi @hmellor @DarkLight1337 , aside from the issue mentioned, are there any discussions about the differences between Sphinx and MkDocs? I couldn’t find any records in GitHub issues or elsewhere, and I’m really curious about the motivations behind migrating between them.

[DarkLight1337]

• MkDocs is more modern than Sphinx (in my opinion)
• MkDocs works natively with Markdown files without the need for extra plugins
• The navigation in MkDocs can be set up in one place which makes it easier to add new sections
• It takes around 5x faster to build the API Docs in MkDocs compared to Sphinx

[gitlawr]

I personally find the previous documentation for vllm serve more straightforward and user-friendly. The available arguments and their explanations were clearly listed, making it easy to check—even on my phone.
https://docs.vllm.ai/en/v0.8.5/serving/openai_compatible_server.html#named-arguments

Now, the documentation only instructs users to run vllm serve --help:
https://docs.vllm.ai/en/stable/configuration/serve_args.html

Or is it expected that we now refer to this page instead?
https://docs.vllm.ai/en/stable/api/vllm/config.html