[Epic] A collection of items related to processing larger than memory datasets (via spilling, externalized algorithm, etc) #14077
Description
Is your feature request related to a problem or challenge?
This epic attempts to organize attempts to improve DataFusion's ability to process datasets that are larger than fit in configured memory budget
Some of DataFusion's "pipeline blocking" operations (SortExec and HashGroupBy) already do work with datasets that are larger than fit in memory, but the performance and usability could be improved
- External sorting not working for (maybe only for string columns??) #12136
- [EPIC] Improved Externalized / Spilling / Large than Memory Hash Aggregation #13123
- Optimized spill file format #14078
- A complete solution for stable and safe sort with spill #14692
- Ensure GroupByHash does not error when trying to spill (calling try_resize where error is not acceptible) #14851
- [EPIC] A collection of tickets for improving sorting larger than memory datasets / spilling sorts #15271
Note: Joins are another operation that can run out of memory and will error (rather than falling back to some other strategy like Sort-Merge-Join for example). If people are interested in making this better, I think we could organize another project
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response