Allow skipping stream buffering #8897
Conversation
| return | ||
|
|
||
| if self.stream_start and chunk_message: | ||
| if getattr(chunk, "_dspy_skip_buffer", None): |
There was a problem hiding this comment.
Per our discussion offline, functionality-wise this has no problem, but this line will for sure confuse our contributors except us two, because _dspy_skip_buffer is not declared/referenced anywhere in the codebase. This looks similar to predict_id handling, but it's entirely internal logic, while this _dspy_skip_buffer requires users to explicitly set it.
I am wondering if it is a better option to just make buffer size adjustable, and let the users handle boilerplate strings. That will be less ideal for internal use cases, while it keeps DSPy unbiased towards broad audience.
There was a problem hiding this comment.
If users need to deal with the boilerplate and want to disable buffering for all chunks, I think users can just use streamify without StreamListener. This option serves for a different case where users (custom LM) know some chunks that don't contain boilerplate.
2 participants
Add logic to check a specific attribute to bypass the streaming buffer