Skip to content

Transforming LogicalPlan::Explain using Treenode::transform fails unexpectedly #8396

@trueleo

Description

@trueleo

Describe the bug

How did i discover this issue

In parseable we have external box for selecting timerange beside query. I want this external filter to be directly applied to any Table referenced in SQL so that this filter expression is available at scan time.

What is the bug ?

In the project I converted the SQL into LogicalPlan and used TreeNode::transform method to mutate the plan to include the external time range filter.

If the plan is Explain, transformation fails with error message. Otherwise it seems to work just fine

 Plan("Invalid EXPLAIN command. Expression is empty")

To Reproduce

minimal program to reproduce this

use std::sync::Arc;
use datafusion::{
    arrow::{
        array::BooleanArray,
        datatypes::{DataType, Field, Fields, Schema},
        record_batch::RecordBatch,
    },
    common::tree_node::{Transformed, TreeNode},
    datasource::{provider_as_source, MemTable},
    logical_expr::{col, Expr, Filter, LogicalPlan, LogicalPlanBuilder},
    scalar::ScalarValue,
};

fn main() {
    // bit of boilerplate code 
    let schema = Arc::new(Schema::new(Fields::from(vec![Field::new(
        "a",
        DataType::Boolean,
        false,
    )])));
    let rb = RecordBatch::try_new(
        schema.clone(),
        vec![Arc::new(BooleanArray::from(vec![true]))],
    )
    .unwrap();
    let table = Arc::new(MemTable::try_new(schema, vec![vec![rb]]).unwrap());

    // example logical plan
    let plan = LogicalPlanBuilder::scan("table", provider_as_source(table), None)
        .unwrap()
        // new explain is created here.
        .explain(false, false)
        .unwrap()
        .build()
        .unwrap();

    // example external filter that i want to add
    let external_filter = col("a").eq(Expr::Literal(ScalarValue::Boolean(Some(true))));

    // after transformation, because plan is not the same anymore,
    // the parent plan is built again with call to LogicalPlan::with_new_inputs -> with_new_exprs
    plan.transform(&|plan| match plan {
        LogicalPlan::TableScan(table) => {
            let filter = Filter::try_new(
                external_filter.clone(),
                Arc::new(LogicalPlan::TableScan(table)),
            )
            .unwrap();
            Ok(Transformed::Yes(LogicalPlan::Filter(filter)))
        }
        x => Ok(Transformed::No(x)),
    })
    // This panics because LogicalPlan::with_new_exprs fail. Explain yeilds empty expression
    // Panic message - Invalid EXPLAIN command. Expression is empty
    .unwrap();

    // run the plan in datafusion session
}

Expected behavior

Creates a new explain plan without any issue.

Additional context

Error orignates in LogicalPlan::new_with_exprs

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions