-
Notifications
You must be signed in to change notification settings - Fork 1k
Closed
Labels
arrowChanges to the arrow crateChanges to the arrow crateenhancementAny new improvement worthy of a entry in the changelogAny new improvement worthy of a entry in the changelog
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Today, DataFusion's expression evaluation does not work natively on DictionaryArrays; Instead what happens is that the DictionaryArrays are unpacked into their base type (e.g. a string dictionary gets unpacked into a StringArray)
This is very inefficient, especially for expressions like col != 'foo' -- if col is a DictionaryArray this filter could be applied by finding 'foo' in the dictionary and then checking for values with that dictionary index, and the dictionary could be reused at the output.
Describe the solution you'd like
- Add kernels that can compare
DictionaryArrays(eq_dict,lt_dict, etc) - Add
DataType::Dictionaryto the theeq_dynand the other dynamic kernels implemented in implementeq_dyn,neq_dyn,lt_dyn,lt_eq_dyn,gt_dyn,gt_eq_dyn#858
Describe alternatives you've considered
N/A
Additional context
Metadata
Metadata
Assignees
Labels
arrowChanges to the arrow crateChanges to the arrow crateenhancementAny new improvement worthy of a entry in the changelogAny new improvement worthy of a entry in the changelog