Streaming RecordBatch support for large datasets (Rust) #855
harshkumar314e
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I’m exploring the possibility of adding streaming query support to ConnectorX (Rust).
Currently, ConnectorX returns all results as a Vec, which is fast for small-to-medium data but inefficient for large datasets — particularly when used inside a backend service where memory is limited.
For example, in my case:
The server should stream results to clients (via Arrow/Parquet/JSON), not buffer all in memory.
Proposed concept
Instead of:
We could expose::
This aligns with Arrow’s RecordBatchStream and similar streaming semantics in DataFusion and Arrow Flight.
Beta Was this translation helpful? Give feedback.
All reactions