Skip to content

Use SimpleExtensions for Substrait type variations #11544

@Blizzara

Description

@Blizzara

Describe the bug

Substrait has an extension mechanism for defining things not included in the protobuf format. See https://substrait.io/extensions/#simple-extensions for definition and https://substrait.io/tutorial/sql_to_substrait/#plans for an example (the example only has scalar functions but idea is the same for types and type variations).

DF has been using the extensions for functions, and #11510 adds support for extension types. However DF still uses self-defined consts for type variations (see https://github.com/apache/datafusion/blob/12d82c427d6c37f7884a508707ccd3058a446908/datafusion/substrait/src/variation_const.rs), rather than writing the variations as extensions in the plan. That makes it hard for other producers/consumers to work with DF variations, since they'd need to match the constants, rather than matching by name (+uri).

We should move type variations to also produce simple extensions.

To Reproduce

No response

Expected behavior

No hardcoded references; type variations written as SimpleExtensionDeclarations à la https://github.com/substrait-io/substrait/blob/a68c1ac62f92d703da624cb8ac0cef854dd2b35f/extensions/type_variations.yaml

Additional context

DF also doesn't use the extensionUris mechanism correctly, the field isn't filled and all extensionUriReference are just hardcoded into max u32. That's filed separately as #11545

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions