Skip to content

Add native comparison kernel support for DictionaryArray #869

@alamb

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Today, DataFusion's expression evaluation does not work natively on DictionaryArrays; Instead what happens is that the DictionaryArrays are unpacked into their base type (e.g. a string dictionary gets unpacked into a StringArray)

This is very inefficient, especially for expressions like col != 'foo' -- if col is a DictionaryArray this filter could be applied by finding 'foo' in the dictionary and then checking for values with that dictionary index, and the dictionary could be reused at the output.

Describe the solution you'd like

  1. Add kernels that can compare DictionaryArrays (eq_dict, lt_dict, etc)
  2. Add DataType::Dictionary to the the eq_dyn and the other dynamic kernels implemented in implement eq_dyn, neq_dyn, lt_dyn, lt_eq_dyn, gt_dyn, gt_eq_dyn #858

Describe alternatives you've considered
N/A

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    arrowChanges to the arrow crateenhancementAny new improvement worthy of a entry in the changelog

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions