docs: Update docs for new UX #2070
Conversation
WalkthroughThis set of changes extensively revises and modernizes documentation throughout the project. The main README and several guides are updated to reflect new installation and usage practices, particularly focusing on direct Python module usage and the Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant PythonModule
participant Engine
participant ClusterService
User->>PythonModule: Run (e.g., python -m dynamo.frontend)
PythonModule->>ClusterService: Connect to etcd, nats
User->>Engine: Install via uv pip install <engine>
PythonModule->>Engine: Launch worker/frontend with specified engine
User->>PythonModule: Send request (e.g., curl)
PythonModule->>Engine: Forward request
Engine-->>PythonModule: Return response
PythonModule-->>User: Stream/return response
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~15–20 minutes Possibly related PRs
Poem
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (5)
docs/architecture/kv_cache_routing.md (2)
9-13: Missing fenced-code language spec (MD040)-``` +```bash python -m dynamo.frontend --router-mode kvAdding
bash(orsh) silences markdown-lint and enables syntax highlighting.
14-15: Minor grammar – stray ‘run’“The Dynamo router run finds …”
Remove “run” for clarity.
-The Dynamo router run finds +The Dynamo router findsREADME.md (1)
65-69: Code block lacks language + wording typo
- Add a language identifier to satisfy MD040.
- “although anyway works” → “although any manager will work”.
-The Dynamo team recommend the `uv` Python package manager, although anyway works. Install uv: -``` +The Dynamo team recommend the `uv` Python package manager, although any manager will work. Install uv: +```bash curl -LsSf https://astral.sh/uv/install.sh | shdocs/guides/backend.md (2)
18-57: Add language identifier to main code sample-``` +```python from dynamo.llm import ModelType, register_llm ... asyncio.run(worker())This fixes MD040 and provides syntax highlighting.
96-100: Typo in example component definition
generat→generate.-Node 2: namespace: llama3-1-8b, component: backend, endpoint: generat, model: /data/Llama-3.1-8B-Instruct/ +Node 2: namespace: llama3-1-8b, component: backend, endpoint: generate, model: /data/Llama-3.1-8B-Instruct/
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (18)
README.md(2 hunks)components/metrics/README.md(1 hunks)docs/API/python_bindings.md(0 hunks)docs/API/sdk.md(0 hunks)docs/architecture/disagg_serving.md(0 hunks)docs/architecture/kv_cache_routing.md(1 hunks)docs/examples/README.md(0 hunks)docs/examples/disagg_skeleton.md(0 hunks)docs/examples/llm_deployment.md(0 hunks)docs/examples/multinode.md(0 hunks)docs/examples/trtllm.md(0 hunks)docs/get_started.md(0 hunks)docs/guides/backend.md(1 hunks)docs/guides/dynamo_deploy/dynamo_operator.md(0 hunks)docs/guides/planner_benchmark/README.md(2 hunks)docs/index.rst(0 hunks)docs/support_matrix.md(0 hunks)lib/llm/src/discovery/watcher.rs(0 hunks)
💤 Files with no reviewable changes (13)
- docs/index.rst
- docs/guides/dynamo_deploy/dynamo_operator.md
- docs/architecture/disagg_serving.md
- docs/support_matrix.md
- docs/examples/README.md
- docs/API/python_bindings.md
- lib/llm/src/discovery/watcher.rs
- docs/get_started.md
- docs/examples/trtllm.md
- docs/examples/disagg_skeleton.md
- docs/examples/multinode.md
- docs/API/sdk.md
- docs/examples/llm_deployment.md
🧰 Additional context used
🧠 Learnings (5)
docs/guides/planner_benchmark/README.md (3)
Learnt from: PeaBrane
PR: #1409
File: examples/router_standalone/worker.py:171-186
Timestamp: 2025-06-08T08:30:45.126Z
Learning: Example code in the examples/ directory may intentionally use hard-coded values or simplified implementations that wouldn't be appropriate for production code, but are acceptable for demonstration and testing purposes.
Learnt from: fsaady
PR: #1730
File: examples/sglang/slurm_jobs/scripts/worker_setup.py:230-244
Timestamp: 2025-07-03T10:14:30.570Z
Learning: In examples/sglang/slurm_jobs/scripts/worker_setup.py, background processes (like nats-server, etcd) are intentionally left running even if later processes fail. This design choice allows users to manually connect to nodes and debug issues without having to restart the entire SLURM job from scratch, providing operational flexibility for troubleshooting in cluster environments.
Learnt from: GuanLuo
PR: #1371
File: examples/llm/benchmarks/vllm_multinode_setup.sh:18-25
Timestamp: 2025-06-05T01:46:15.509Z
Learning: In multi-node setups with head/worker architecture, the head node typically doesn't need environment variables pointing to its own services (like NATS_SERVER, ETCD_ENDPOINTS) because local processes can access them via localhost. Only worker nodes need these environment variables to connect to the head node's external IP address.
components/metrics/README.md (2)
Learnt from: PeaBrane
PR: #1392
File: launch/dynamo-run/src/subprocess/vllm_v1_inc.py:71-71
Timestamp: 2025-06-05T01:04:24.775Z
Learning: The create_endpoint method in WorkerMetricsPublisher has backward compatibility maintained through pyo3 signature annotation #[pyo3(signature = (component, dp_rank = None))], making the dp_rank parameter optional with a default value of None.
Learnt from: biswapanda
PR: #1890
File: examples/vllm/deploy/agg.yaml:63-70
Timestamp: 2025-07-14T23:01:16.218Z
Learning: In vLLM worker deployments, grep-based log checks for "VllmWorker.*has been initialized" are appropriate for readiness probes to verify worker startup, but should not be used for liveness probes which need to detect ongoing worker health.
docs/architecture/kv_cache_routing.md (1)
Learnt from: PeaBrane
PR: #1285
File: lib/llm/src/kv_router/scheduler.rs:260-266
Timestamp: 2025-05-30T06:34:12.785Z
Learning: In the KV router scheduler code, PeaBrane prefers fail-fast behavior over silent failure handling. When accessing worker metrics data that could be out-of-bounds (like dp_rank indexing), explicit panics are preferred over graceful degradation with continue statements to ensure data integrity issues are caught early.
README.md (3)
Learnt from: biswapanda
PR: #1412
File: lib/bindings/python/src/dynamo/runtime/logging.py:100-100
Timestamp: 2025-06-06T21:48:35.214Z
Learning: In the Dynamo codebase, BentoML has been completely removed from all executable code, with only documentation and attribution references remaining. The error_loggers configuration in lib/bindings/python/src/dynamo/runtime/logging.py should not include "bentoml" since those modules no longer exist.
Learnt from: ptarasiewiczNV
PR: #2027
File: container/deps/vllm/install_vllm.sh:0-0
Timestamp: 2025-07-22T10:22:28.972Z
Learning: The --torch-backend=auto flag works with vLLM installations via uv pip install, even though it's not a standard pip option. This flag is processed by vLLM's build system during installation to automatically match PyTorch distribution with container CUDA versions.
Learnt from: fsaady
PR: #1730
File: examples/sglang/slurm_jobs/scripts/worker_setup.py:230-244
Timestamp: 2025-07-03T10:14:30.570Z
Learning: In examples/sglang/slurm_jobs/scripts/worker_setup.py, background processes (like nats-server, etcd) are intentionally left running even if later processes fail. This design choice allows users to manually connect to nodes and debug issues without having to restart the entire SLURM job from scratch, providing operational flexibility for troubleshooting in cluster environments.
docs/guides/backend.md (5)
Learnt from: nnshah1
PR: #1444
File: tests/fault_tolerance/utils/metrics.py:30-32
Timestamp: 2025-07-01T13:55:03.940Z
Learning: The @dynamo_worker() decorator in the dynamo codebase returns a wrapper that automatically injects the runtime parameter before calling the wrapped function. This means callers only need to provide the non-runtime parameters, while the decorator handles injecting the runtime argument automatically. For example, a function with signature async def get_metrics(runtime, log_dir) decorated with @dynamo_worker() can be called as get_metrics(log_dir) because the decorator wrapper injects the runtime parameter.
Learnt from: PeaBrane
PR: #1392
File: launch/dynamo-run/src/subprocess/vllm_v1_inc.py:71-71
Timestamp: 2025-06-05T01:04:24.775Z
Learning: The create_endpoint method in WorkerMetricsPublisher has backward compatibility maintained through pyo3 signature annotation #[pyo3(signature = (component, dp_rank = None))], making the dp_rank parameter optional with a default value of None.
Learnt from: biswapanda
PR: #1412
File: lib/bindings/python/src/dynamo/runtime/logging.py:100-100
Timestamp: 2025-06-06T21:48:35.214Z
Learning: In the Dynamo codebase, BentoML has been completely removed from all executable code, with only documentation and attribution references remaining. The error_loggers configuration in lib/bindings/python/src/dynamo/runtime/logging.py should not include "bentoml" since those modules no longer exist.
Learnt from: GuanLuo
PR: #1371
File: examples/llm/benchmarks/vllm_multinode_setup.sh:18-25
Timestamp: 2025-06-05T01:46:15.509Z
Learning: In multi-node setups with head/worker architecture, the head node typically doesn't need environment variables pointing to its own services (like NATS_SERVER, ETCD_ENDPOINTS) because local processes can access them via localhost. Only worker nodes need these environment variables to connect to the head node's external IP address.
Learnt from: fsaady
PR: #1730
File: examples/sglang/slurm_jobs/scripts/worker_setup.py:230-244
Timestamp: 2025-07-03T10:14:30.570Z
Learning: In examples/sglang/slurm_jobs/scripts/worker_setup.py, background processes (like nats-server, etcd) are intentionally left running even if later processes fail. This design choice allows users to manually connect to nodes and debug issues without having to restart the entire SLURM job from scratch, providing operational flexibility for troubleshooting in cluster environments.
🪛 markdownlint-cli2 (0.17.2)
docs/architecture/kv_cache_routing.md
10-10: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
README.md
66-66: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
74-74: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
91-91: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
96-96: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
105-105: Multiple headings with the same content
(MD024, no-duplicate-heading)
113-113: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
148-148: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
153-153: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
163-163: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
168-168: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
172-172: Bare URL used
(MD034, no-bare-urls)
181-181: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
186-186: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
193-193: Bare URL used
(MD034, no-bare-urls)
196-196: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
202-202: Heading levels should only increment by one level at a time
Expected: h2; Actual: h3
(MD001, heading-increment)
207-207: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
213-213: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
219-219: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
230-230: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
237-237: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
244-244: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
252-252: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
259-259: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🪛 LanguageTool
README.md
[grammar] ~65-~65: Ensure spelling is correct
Context: ...e uv Python package manager, although anyway works. Install uv: ``` curl -LsSf https...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Build and Test - vllm
- GitHub Check: pre-merge-rust (.)
🔇 Additional comments (1)
components/metrics/README.md (1)
78-85: Code sample drops required env-vars / component name may be wrong
- A local deployment still needs
ETCD_ENDPOINTSandNATS_SERVERexported, otherwisepython -m dynamo.frontendwill fail to start.- The metrics command now hard-codes
--component backend; this only works if the worker registered under that exact component name. Please call this out explicitly or use an example that matches the preceding worker command (VllmWorker→backend).
biswapanda
left a comment
biswapanda
left a comment
There was a problem hiding this comment.
Lgtm.
rust 1 line change looks fine and docs lgtm at high level
Work in progress
Because if not get_started, what is the README meant to help you do?
Uh oh!
There was an error while loading. Please reload this page.