[substrait] [sqllogictest] Null cast is not valid

[gabotechs]

Describe the bug

~700 sqllogictest cases are failing in Substrait round-trip mode because of the following error:

query failed: DataFusion error: Internal error: Null cast is not valid.

To Reproduce

cargo test --test sqllogictests -- --substrait-round-trip aggregate_skip_partial.slt:194

Expected behavior

The sqllogictest passes in Substrait round-trip mode

Additional context

This is part of a broader initiative to improve the reliability of Substrait-to-DataFusion logical plan conversions.

• [Epic] A collection of Substrait conversion issues #16248

[jkosh44]

take

[jkosh44]

Here's a more minimal .slt repro

query I rowsort
select column1 from VALUES (1) where column1 = NULL
----

[jkosh44]

The failure happens when converting the DF logical plan into substrait. The logical plan of the above query looks like

Projection(Projection {
    expr: [
        Column(Column {
            relation: None,
            name: "column1"
        })
    ],
    input: Filter(Filter {
        predicate: BinaryExpr(BinaryExpr {
            left: Column(Column {
                relation: None,
                name: "column1"
            }),
            op: Eq,
            right: Literal(NULL, None)
        }),
        input: Values(Values {
            schema: DFSchema {
                inner: Schema {
                    fields: [
                        Field {
                            name: "column1",
                            data_type: Int64,
                            nullable: true,
                            dict_id: 0,
                            dict_is_ordered: false,
                            metadata: {}
                        }
                    ],
                    metadata: {}
                },
                field_qualifiers: [None],
                functional_dependencies: FunctionalDependencies { deps: [] }
            },
            values: [
                [Literal(Int64(1), None)]
            ]
        })
    }),
    schema: DFSchema {
        inner: Schema {
            fields: [
                Field {
                    name: "column1",
                    data_type: Int64,
                    nullable: true,
                    dict_id: 0,
                    dict_is_ordered: false,
                    metadata: {}
                }
            ],
            metadata: {}
        },
        field_qualifiers: [None],
        functional_dependencies: FunctionalDependencies { deps: [] }
    }
})

The conversion that fails is the right hand side of the predicate, specifically right: Literal(NULL, None). We correctly identify that the literal is null.

datafusion/datafusion/substrait/src/logical_plan/producer/expr/literal.rs

Lines 54 to 67 in 8f15991

	pub(crate) fn to_substrait_literal(
	producer: &mut impl SubstraitProducer,
	value: &ScalarValue,
	) -> datafusion::common::Result<Literal> {
	if value.is_null() {
	return Ok(Literal {
	nullable: true,
	type_variation_reference: DEFAULT_TYPE_VARIATION_REF,
	literal_type: Some(LiteralType::Null(to_substrait_type(
	&value.data_type(),
	true,
	)?)),
	});
	}

However, when we see that the type of the null is NULL, we return an error.

datafusion/datafusion/substrait/src/logical_plan/producer/types.rs

Lines 34 to 44 in 8f15991

	pub(crate) fn to_substrait_type(
	dt: &DataType,
	nullable: bool,
	) -> datafusion::common::Result<substrait::proto::Type> {
	let nullability = if nullable {
	r#type::Nullability::Nullable as i32
	} else {
	r#type::Nullability::Required as i32
	};
	match dt {
	DataType::Null => internal_err!("Null cast is not valid"),

I think for this to work properly, we have to know the actual type of the null, since substrait nulls must be typed. For example if we modify the .slt to the following

query I rowsort
select column1 from VALUES (1) where column1 = (NULL::INT)
----

it works fine.

[jkosh44]

I can't figure out how to un-assign myself but I don't think I have the time to continue on this. For whoever picks this up here are some ideas about how to resolve this

• Add support for untyped NULL to substrait.
• Perform a pass over the Datafusion logical plan to add type information to untyped NULLs. It's possible that this is already implemented elsewhere in Datafusion and then we can just re-use that logic. Otherwise I think it will be a bit non-trivial to implement this in general.

[gene-bordegaray]

take