feat: Enable disagg support in trtllm standalone script #1355
Conversation
tanmayv25
requested review from
a team,
GuanLuo,
PeaBrane,
alec-flowers,
biswapanda,
grahamking,
ishandhanani,
jthomson04,
kkranen,
nnshah1,
paulhendricks,
piotrm-nvidia,
ptarasiewiczNV,
rmccorm4,
ryanolson,
tedzhouhk and
tmonty12
as code owners
WalkthroughThis update introduces a disaggregated serving architecture to the TRTLLM Dynamo integration, supporting "prefill", "decode", and "prefill_and_decode" modes. It adds a sample YAML configuration for GPU memory management and implements remote prefill requests, base64 encoding/decoding of disaggregated parameters, new command-line options, endpoint parsing, and conditional initialization and request handling based on the selected mode. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant DecodeWorker
participant PrefillWorker
Client->>DecodeWorker: Send generation request (decode mode)
DecodeWorker->>PrefillWorker: Remote prefill request (context_only, max_tokens=1)
PrefillWorker-->>DecodeWorker: Return prefill result (encoded params/tokens)
alt Prefill indicates stop/error
DecodeWorker-->>Client: Yield stop/error response
else Continue generation
DecodeWorker->>DecodeWorker: Update params to generation_only
DecodeWorker-->>Client: Yield first token, continue generation
end
Poem
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🔭 Outside diff range comments (1)
launch/dynamo-run/src/subprocess/trtllm_inc.py (1)
309-315:⚠️ Potential issue
kv_cache_configtreated as object, but it is a dict – will raiseAttributeError
kv_cache_config.event_buffer_max_sizeuses attribute access on a dictionary:kv_cache_config = arg_map["kv_cache_config"] if not kv_cache_config.event_buffer_max_size: kv_cache_config.event_buffer_max_size = DEFAULT_KV_EVENT_BUFFER_MAX_SIZEThis will crash at startup. Replace with key access:
-if not kv_cache_config.event_buffer_max_size: - kv_cache_config.event_buffer_max_size = DEFAULT_KV_EVENT_BUFFER_MAX_SIZE +if not kv_cache_config.get("event_buffer_max_size"): + kv_cache_config["event_buffer_max_size"] = DEFAULT_KV_EVENT_BUFFER_MAX_SIZE
🧹 Nitpick comments (2)
launch/dynamo-run/src/subprocess/trtllm_inc.py (1)
80-87:DisaggregatedParamsCodecmutates the input object in-place
encode()anddecode()overwriteopaque_stateon the same dataclass instance that upstream code may still hold a reference to, creating surprising side-effects and potential data races if the instance is reused concurrently.A safer pattern is to return a new instance:
- disaggregated_params.opaque_state = opaque_state - return disaggregated_params + return DisaggregatedParams( + request_type=disaggregated_params.request_type, + opaque_state=opaque_state, + )Apply the same pattern to
encode().
While here, theelse:branches are unnecessary after an earlyreturn.Also applies to: 95-101
launch/dynamo-run/src/subprocess/trtllm_config/sample.yaml (1)
22-24: Minor style nit – avoid trailing zeros unless requiredYAML parses
0.40as the float0.4, so the extra trailing zero does not affect behaviour.
Removing it keeps the file consistent with common YAML style guides:- free_gpu_memory_fraction: 0.40 + free_gpu_memory_fraction: 0.4
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
launch/dynamo-run/src/subprocess/trtllm_config/sample.yaml(1 hunks)launch/dynamo-run/src/subprocess/trtllm_inc.py(12 hunks)
🧰 Additional context used
🪛 Pylint (3.3.7)
launch/dynamo-run/src/subprocess/trtllm_inc.py
[convention] 13-13: Line too long (107/100)
(C0301)
[convention] 14-14: Line too long (109/100)
(C0301)
[error] 27-27: Unable to import 'uvloop'
(E0401)
[error] 30-30: Unable to import 'tensorrt_llm'
(E0401)
[error] 31-31: Unable to import 'tensorrt_llm.llmapi'
(E0401)
[convention] 61-61: Line too long (134/100)
(C0301)
[convention] 56-56: Missing function or method docstring
(C0116)
[warning] 60-62: Use lazy % formatting in logging functions
(W1203)
[convention] 74-74: Missing function or method docstring
(C0116)
[refactor] 77-86: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it
(R1705)
[convention] 89-89: Missing function or method docstring
(C0116)
[refactor] 92-101: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it
(R1705)
[convention] 156-156: Line too long (151/100)
(C0301)
[convention] 165-165: Line too long (121/100)
(C0301)
[warning] 156-156: TODO: Use smart KV router to determine which prefill worker to use. This would also require supporting publishing events for prefill workers.
(W0511)
[refactor] 125-125: Too many arguments (7/5)
(R0913)
[refactor] 125-125: Too many positional arguments (7/5)
(R0917)
[convention] 142-142: Missing function or method docstring
(C0116)
[warning] 275-277: Use lazy % formatting in logging functions
(W1203)
[warning] 343-343: FIXME: Enable publishing events and metrics for disaggregated prefill.
(W0511)
[convention] 436-436: Line too long (141/100)
(C0301)
[convention] 443-443: Line too long (152/100)
(C0301)
[convention] 449-449: Line too long (181/100)
(C0301)
[convention] 456-456: Line too long (221/100)
(C0301)
[convention] 464-464: Line too long (105/100)
(C0301)
[convention] 472-472: Line too long (109/100)
(C0301)
⏰ Context from checks skipped due to timeout of 90000ms (3)
- GitHub Check: Build and Test - vllm
- GitHub Check: pre-merge-rust (lib/bindings/python)
- GitHub Check: pre-merge-rust (.)
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
launch/dynamo-run/src/subprocess/trtllm_inc.py (1)
182-187:⚠️ Potential issueCritical async generator handling issue still present
The code is still awaiting
round_robin()before usingasync for, which was flagged in a previous review. This will cause aTypeErrorifround_robin()returns an async generator.Apply this fix to remove the
awaitbefore theasync forloop:- remote_prefill_responses = [ - remote_prefill_response - async for remote_prefill_response in await self.remote_prefill_client.round_robin( - prefill_request - ) - ] + remote_prefill_responses = [ + remote_prefill_response + async for remote_prefill_response in self.remote_prefill_client.round_robin( + prefill_request + ) + ]
🧹 Nitpick comments (4)
launch/dynamo-run/src/subprocess/trtllm_inc.py (4)
68-112: Consider optimizing base64 encoding/decodingThe
DisaggregatedParamsCodecclass handles base64 encoding/decoding correctly, but there are some style improvements that could be made.Apply this diff to remove unnecessary else statements and improve readability:
@staticmethod def decode( disaggregated_params: DisaggregatedParams, ) -> DisaggregatedParams: if disaggregated_params is None: return None - else: - opaque_state = ( - base64.b64decode(disaggregated_params.opaque_state) - if disaggregated_params.opaque_state is not None - else None - ) - return DisaggregatedParams( - request_type=disaggregated_params.request_type, - first_gen_tokens=disaggregated_params.first_gen_tokens, - ctx_request_id=disaggregated_params.ctx_request_id, - opaque_state=opaque_state, - draft_tokens=disaggregated_params.draft_tokens, - ) + + opaque_state = ( + base64.b64decode(disaggregated_params.opaque_state) + if disaggregated_params.opaque_state is not None + else None + ) + return DisaggregatedParams( + request_type=disaggregated_params.request_type, + first_gen_tokens=disaggregated_params.first_gen_tokens, + ctx_request_id=disaggregated_params.ctx_request_id, + opaque_state=opaque_state, + draft_tokens=disaggregated_params.draft_tokens, + ) @staticmethod def encode( disaggregated_params: DisaggregatedParams, ) -> DisaggregatedParams: if disaggregated_params is None: return None - else: - encoded_opaque_state = ( - base64.b64encode(disaggregated_params.opaque_state).decode("utf-8") - if disaggregated_params.opaque_state is not None - else None - ) - return DisaggregatedParams( - request_type=disaggregated_params.request_type, - first_gen_tokens=disaggregated_params.first_gen_tokens, - ctx_request_id=disaggregated_params.ctx_request_id, - opaque_state=encoded_opaque_state, - draft_tokens=disaggregated_params.draft_tokens, - ) + + encoded_opaque_state = ( + base64.b64encode(disaggregated_params.opaque_state).decode("utf-8") + if disaggregated_params.opaque_state is not None + else None + ) + return DisaggregatedParams( + request_type=disaggregated_params.request_type, + first_gen_tokens=disaggregated_params.first_gen_tokens, + ctx_request_id=disaggregated_params.ctx_request_id, + opaque_state=encoded_opaque_state, + draft_tokens=disaggregated_params.draft_tokens, + )🧰 Tools
🪛 Pylint (3.3.7)
[convention] 74-74: Missing function or method docstring
(C0116)
[refactor] 77-91: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it
(R1705)
[convention] 94-94: Missing function or method docstring
(C0116)
[refactor] 97-111: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it
(R1705)
150-158: Consider refactoring constructor with too many parametersThe
RequestHandlerconstructor now accepts 7 parameters, which exceeds the typical limit of 5. Consider using a configuration object or builder pattern to improve maintainability.Consider creating a
RequestHandlerConfigdataclass:from dataclasses import dataclass @dataclass class RequestHandlerConfig: component: object engine: object default_sampling_params: object publishers: object disaggregation_mode: str remote_prefill_client: object class RequestHandler: def __init__(self, config: RequestHandlerConfig): self.engine = config.engine self.component = config.component # ... etc🧰 Tools
🪛 Pylint (3.3.7)
[refactor] 150-150: Too many arguments (7/5)
(R0913)
[refactor] 150-150: Too many positional arguments (7/5)
(R0917)
181-181: Address TODO comment for production readinessThe TODO comment about implementing a smart KV router is critical for production deployments. The current round-robin approach may not be optimal for load distribution.
Do you want me to create an issue to track the implementation of the smart KV router functionality, or would you like me to propose a basic implementation approach?
🧰 Tools
🪛 Pylint (3.3.7)
[convention] 181-181: Line too long (151/100)
(C0301)
[warning] 181-181: TODO: Use smart KV router to determine which prefill worker to use. This would also require supporting publishing events for prefill workers.
(W0511)
370-371: Address FIXME for disaggregated prefill metricsThe FIXME comment indicates that event and metrics publishing isn't supported for disaggregated prefill workers, which could impact observability in production.
This limitation could affect monitoring and debugging in production environments. Do you want me to create an issue to track enabling metrics for prefill workers, or would you like assistance implementing this functionality?
🧰 Tools
🪛 Pylint (3.3.7)
[warning] 370-370: FIXME: Enable publishing events and metrics for disaggregated prefill.
(W0511)
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
launch/dynamo-run/src/subprocess/trtllm_inc.py(12 hunks)
🧰 Additional context used
🪛 Pylint (3.3.7)
launch/dynamo-run/src/subprocess/trtllm_inc.py
[convention] 14-14: Line too long (102/100)
(C0301)
[error] 27-27: Unable to import 'uvloop'
(E0401)
[error] 30-30: Unable to import 'tensorrt_llm'
(E0401)
[error] 31-31: Unable to import 'tensorrt_llm.llmapi'
(E0401)
[convention] 56-56: Missing function or method docstring
(C0116)
[convention] 74-74: Missing function or method docstring
(C0116)
[refactor] 77-91: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it
(R1705)
[convention] 94-94: Missing function or method docstring
(C0116)
[refactor] 97-111: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it
(R1705)
[convention] 181-181: Line too long (151/100)
(C0301)
[convention] 190-190: Line too long (121/100)
(C0301)
[warning] 181-181: TODO: Use smart KV router to determine which prefill worker to use. This would also require supporting publishing events for prefill workers.
(W0511)
[refactor] 150-150: Too many arguments (7/5)
(R0913)
[refactor] 150-150: Too many positional arguments (7/5)
(R0917)
[convention] 167-167: Missing function or method docstring
(C0116)
[warning] 298-298: Use lazy % formatting in logging functions
(W1203)
[warning] 302-304: Use lazy % formatting in logging functions
(W1203)
[warning] 370-370: FIXME: Enable publishing events and metrics for disaggregated prefill.
(W0511)
[convention] 463-463: Line too long (141/100)
(C0301)
[convention] 471-471: Line too long (158/100)
(C0301)
[convention] 477-477: Line too long (181/100)
(C0301)
[convention] 484-484: Line too long (221/100)
(C0301)
[convention] 506-506: Line too long (105/100)
(C0301)
[convention] 514-514: Line too long (109/100)
(C0301)
⏰ Context from checks skipped due to timeout of 90000ms (3)
- GitHub Check: pre-merge-rust (lib/bindings/python)
- GitHub Check: pre-merge-rust (.)
- GitHub Check: Build and Test - vllm
🔇 Additional comments (2)
launch/dynamo-run/src/subprocess/trtllm_inc.py (2)
56-66: LGTM: Exception handling improvedThe
parse_endpointfunction now correctly raisesValueErrorinstead of callingsys.exit(1), addressing the previous review feedback. This allows callers to handle errors gracefully.🧰 Tools
🪛 Pylint (3.3.7)
[convention] 56-56: Missing function or method docstring
(C0116)
490-517: LGTM: Robust task validation and mode handlingThe disaggregation mode validation logic correctly ensures only one task is specified and handles the prefill mode restrictions appropriately. The warnings for incompatible options are helpful for users.
🧰 Tools
🪛 Pylint (3.3.7)
[convention] 506-506: Line too long (105/100)
(C0301)
[convention] 514-514: Line too long (109/100)
(C0301)
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
launch/dynamo-run/src/subprocess/trtllm_inc.py (1)
173-221: Fix the await/async for pattern in remote_prefill.The method still has the same issue flagged in past reviews - using
awaitbeforeasync forwill cause a runtime TypeError ifround_robinreturns an async generator.Apply this fix:
- async for remote_prefill_response in await self.remote_prefill_client.round_robin( - prefill_request - ) + async for remote_prefill_response in self.remote_prefill_client.round_robin( + prefill_request + )🧰 Tools
🪛 Pylint (3.3.7)
[convention] 201-201: Line too long (155/100)
(C0301)
[convention] 213-213: Line too long (121/100)
(C0301)
[warning] 201-201: TODO: Use smart KV router to determine which prefill worker to use. This would also require supporting publishing events for prefill workers.
(W0511)
[warning] 209-209: Consider explicitly re-raising using 'raise ValueError(f'Error in remote prefill: {e}') from e'
(W0707)
🧹 Nitpick comments (5)
launch/dynamo-run/src/subprocess/trtllm_inc.py (5)
209-209: Improve exception chaining for better debugging.Use explicit exception chaining to preserve the original stack trace when re-raising exceptions.
Apply these fixes:
- raise ValueError(f"Error in remote prefill: {e}") + raise ValueError(f"Error in remote prefill: {e}") from e- raise ValueError(f"Error in remote prefill: {e}") + raise ValueError(f"Error in remote prefill: {e}") from eAlso applies to: 247-247
🧰 Tools
🪛 Pylint (3.3.7)
[warning] 209-209: Consider explicitly re-raising using 'raise ValueError(f'Error in remote prefill: {e}') from e'
(W0707)
222-317: Consider refactoring the complex generate method.The
generatemethod has become quite complex with multiple branches and responsibilities (18 local variables, 16 branches, 53 statements according to static analysis). Consider extracting some logic into helper methods for better maintainability.Potential refactoring opportunities:
- Extract disaggregated parameter handling logic
- Extract remote prefill response processing
- Extract output token formatting logic
This would improve readability and make the method easier to test and maintain.
🧰 Tools
🪛 Pylint (3.3.7)
[warning] 282-282: TODO: Disable streaming for context only requests when adding disagg support
(W0511)
[convention] 222-222: Missing function or method docstring
(C0116)
[refactor] 222-222: Too many local variables (18/15)
(R0914)
[warning] 247-247: Consider explicitly re-raising using 'raise ValueError(f'Error in remote prefill: {e}') from e'
(W0707)
[refactor] 222-222: Too many branches (16/12)
(R0912)
[refactor] 222-222: Too many statements (53/50)
(R0915)
329-329: Use lazy formatting in logging statements.For better performance, use lazy % formatting instead of f-strings in logging calls.
Apply these fixes:
- logging.info(f"Initializing the worker with config: {config}") + logging.info("Initializing the worker with config: %s", config)- logging.info( - f"Initializing remote prefill client for endpoint: {config.remote_prefill_endpoint}" - ) + logging.info( + "Initializing remote prefill client for endpoint: %s", config.remote_prefill_endpoint + )Also applies to: 333-335
🧰 Tools
🪛 Pylint (3.3.7)
[warning] 329-329: Use lazy % formatting in logging functions
(W1203)
201-201: Address line length violations.Several lines exceed the 100-character limit. Consider breaking them for better readability.
Examples of lines that could be shortened:
- Line 201: Break the TODO comment across multiple lines
- Line 213: Break the error message string
- Lines 490+: Break long help strings using implicit string concatenation
Also applies to: 213-213, 490-490, 498-498, 504-504, 511-511, 533-533, 541-541
🧰 Tools
🪛 Pylint (3.3.7)
[convention] 201-201: Line too long (155/100)
(C0301)
[warning] 201-201: TODO: Use smart KV router to determine which prefill worker to use. This would also require supporting publishing events for prefill workers.
(W0511)
201-201: Track TODO and FIXME items for future development.The code contains important TODO and FIXME comments that should be tracked:
- Line 201: Smart KV router implementation for prefill worker selection
- Line 401: Enable publishing events and metrics for disaggregated prefill
These represent important future enhancements for the disaggregated serving architecture. Would you like me to help create GitHub issues to track these items?
Also applies to: 401-401
🧰 Tools
🪛 Pylint (3.3.7)
[convention] 201-201: Line too long (155/100)
(C0301)
[warning] 201-201: TODO: Use smart KV router to determine which prefill worker to use. This would also require supporting publishing events for prefill workers.
(W0511)
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
launch/dynamo-run/src/subprocess/trtllm_inc.py(11 hunks)
🧰 Additional context used
🪛 Pylint (3.3.7)
launch/dynamo-run/src/subprocess/trtllm_inc.py
[convention] 14-14: Line too long (102/100)
(C0301)
[error] 27-27: Unable to import 'uvloop'
(E0401)
[error] 30-30: Unable to import 'tensorrt_llm'
(E0401)
[error] 31-31: Unable to import 'tensorrt_llm.llmapi'
(E0401)
[convention] 56-56: Missing function or method docstring
(C0116)
[convention] 74-74: Missing function or method docstring
(C0116)
[convention] 94-94: Missing function or method docstring
(C0116)
[convention] 201-201: Line too long (155/100)
(C0301)
[convention] 213-213: Line too long (121/100)
(C0301)
[warning] 201-201: TODO: Use smart KV router to determine which prefill worker to use. This would also require supporting publishing events for prefill workers.
(W0511)
[warning] 209-209: Consider explicitly re-raising using 'raise ValueError(f'Error in remote prefill: {e}') from e'
(W0707)
[convention] 222-222: Missing function or method docstring
(C0116)
[refactor] 222-222: Too many local variables (18/15)
(R0914)
[warning] 247-247: Consider explicitly re-raising using 'raise ValueError(f'Error in remote prefill: {e}') from e'
(W0707)
[refactor] 222-222: Too many branches (16/12)
(R0912)
[refactor] 222-222: Too many statements (53/50)
(R0915)
[warning] 329-329: Use lazy % formatting in logging functions
(W1203)
[warning] 333-335: Use lazy % formatting in logging functions
(W1203)
[warning] 401-401: FIXME: Enable publishing events and metrics for disaggregated prefill.
(W0511)
[convention] 490-490: Line too long (141/100)
(C0301)
[convention] 498-498: Line too long (158/100)
(C0301)
[convention] 504-504: Line too long (181/100)
(C0301)
[convention] 511-511: Line too long (221/100)
(C0301)
[convention] 533-533: Line too long (105/100)
(C0301)
[convention] 541-541: Line too long (109/100)
(C0301)
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: pre-merge-rust (.)
- GitHub Check: Build and Test - vllm
🔇 Additional comments (8)
launch/dynamo-run/src/subprocess/trtllm_inc.py (8)
10-15: Clear documentation of disaggregated serving usage patterns.The added documentation provides helpful examples of how to launch the different components in disaggregated mode.
🧰 Tools
🪛 Pylint (3.3.7)
[convention] 14-14: Line too long (102/100)
(C0301)
56-66: Good refactoring of endpoint parsing logic.The extraction of
parse_endpointinto a standalone function improves reusability and the error handling has been properly updated to raiseValueErrorinstead of callingsys.exit()as suggested in past reviews.🧰 Tools
🪛 Pylint (3.3.7)
[convention] 56-56: Missing function or method docstring
(C0116)
68-112: Well-designed codec for network transfer of disaggregated parameters.The
DisaggregatedParamsCodecclass provides a clean abstraction for base64 encoding/decoding of opaque state data for network transfer between workers. The symmetric encode/decode operations and null-safety checks are implemented correctly.🧰 Tools
🪛 Pylint (3.3.7)
[convention] 74-74: Missing function or method docstring
(C0116)
[convention] 94-94: Missing function or method docstring
(C0116)
126-143: Enhanced Config class with useful debugging output.The addition of disaggregation fields and the
__str__method improves debugging capabilities and configuration transparency.
145-157: Good separation of concerns with RequestHandlerConfig.The new dataclass encapsulates request handler configuration cleanly, improving code organization.
397-410: Good conditional logic for disaggregated mode handling.The conditional model registration based on disaggregation mode is correctly implemented. Prefill workers appropriately skip registration since they receive requests directly from decode workers.
🧰 Tools
🪛 Pylint (3.3.7)
[warning] 401-401: FIXME: Enable publishing events and metrics for disaggregated prefill.
(W0511)
492-505: Well-designed command line interface for disaggregation modes.The new
--taskand--remote-prefill-endpointarguments provide a clean interface for configuring disaggregated serving modes with appropriate validation.🧰 Tools
🪛 Pylint (3.3.7)
[convention] 498-498: Line too long (158/100)
(C0301)
[convention] 504-504: Line too long (181/100)
(C0301)
517-544: Robust validation logic for task conflicts and mode-specific constraints.The validation logic correctly handles conflicting tasks and enforces mode-specific constraints (e.g., remote prefill endpoint restrictions for prefill mode).
🧰 Tools
🪛 Pylint (3.3.7)
[convention] 533-533: Line too long (105/100)
(C0301)
[convention] 541-541: Line too long (109/100)
(C0301)
Overview:
Adds support for launching workers in disaggregated mode via dynamo run for TRTLLM. This is a short-term support. We will add more updates in dynamo ingress to support pipelining stages.
Details:
There are three disagg modes for the engine: none, prefill and decode. decode worker listens to the ingress and publishes the model card. decode worker directly talks to prefill worker via client interface.
The user can start these services directly and send oai requests to the ingress:
They can launch multiple such workers on different nodes.
Status:
Pending:
Where should the reviewer start?
trtllm_inc.py is the file that is updated.
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
N/A
Summary by CodeRabbit
New Features
Bug Fixes
Documentation