-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
How did i discover this issue
In parseable we have external box for selecting timerange beside query. I want this external filter to be directly applied to any Table referenced in SQL so that this filter expression is available at scan time.
What is the bug ?
In the project I converted the SQL into LogicalPlan and used TreeNode::transform method to mutate the plan to include the external time range filter.
If the plan is Explain, transformation fails with error message. Otherwise it seems to work just fine
Plan("Invalid EXPLAIN command. Expression is empty")
To Reproduce
minimal program to reproduce this
use std::sync::Arc;
use datafusion::{
arrow::{
array::BooleanArray,
datatypes::{DataType, Field, Fields, Schema},
record_batch::RecordBatch,
},
common::tree_node::{Transformed, TreeNode},
datasource::{provider_as_source, MemTable},
logical_expr::{col, Expr, Filter, LogicalPlan, LogicalPlanBuilder},
scalar::ScalarValue,
};
fn main() {
// bit of boilerplate code
let schema = Arc::new(Schema::new(Fields::from(vec![Field::new(
"a",
DataType::Boolean,
false,
)])));
let rb = RecordBatch::try_new(
schema.clone(),
vec![Arc::new(BooleanArray::from(vec![true]))],
)
.unwrap();
let table = Arc::new(MemTable::try_new(schema, vec![vec![rb]]).unwrap());
// example logical plan
let plan = LogicalPlanBuilder::scan("table", provider_as_source(table), None)
.unwrap()
// new explain is created here.
.explain(false, false)
.unwrap()
.build()
.unwrap();
// example external filter that i want to add
let external_filter = col("a").eq(Expr::Literal(ScalarValue::Boolean(Some(true))));
// after transformation, because plan is not the same anymore,
// the parent plan is built again with call to LogicalPlan::with_new_inputs -> with_new_exprs
plan.transform(&|plan| match plan {
LogicalPlan::TableScan(table) => {
let filter = Filter::try_new(
external_filter.clone(),
Arc::new(LogicalPlan::TableScan(table)),
)
.unwrap();
Ok(Transformed::Yes(LogicalPlan::Filter(filter)))
}
x => Ok(Transformed::No(x)),
})
// This panics because LogicalPlan::with_new_exprs fail. Explain yeilds empty expression
// Panic message - Invalid EXPLAIN command. Expression is empty
.unwrap();
// run the plan in datafusion session
}Expected behavior
Creates a new explain plan without any issue.
Additional context
Error orignates in LogicalPlan::new_with_exprs
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working