[EPIC] Tracking issue of support substrait logical plan #8149
Description
Is your feature request related to a problem or challenge?
Tracks substrait support progress
The rough goal of this ticket is:
- Support all basic operations that datafusion supports. I.e., reach the same coverage and functionalities as
datafusion-proto - Can co-operate with one or two external systems, like
spark-substrait.
Describe the solution you'd like
LogicalPlan
- CrossJoin Support crossjoin in substrait. #8427
- Repartition feat: implement Repartition plan in substrait #8526
- Subquery
- Values & EmptyRelation Support Substrait's VirtualTables #10531
- Unnest
The following logical plans I think we don't need to support:
- Statement
- Analyze
- Explain
- Prepare
- Dml
- Ddl
- Copy
- DescribeTable
Expression
- Negative feat: implement Unary Expr in substrait #8534
- SimilarTo
- Not feat: implement Unary Expr in substrait #8534
- IsTrue feat: implement Unary Expr in substrait #8534
- IsFalse feat: implement Unary Expr in substrait #8534
- IsUnknown feat: implement Unary Expr in substrait #8534
- IsNotTrue feat: implement Unary Expr in substrait #8534
- IsNotFalse feat: implement Unary Expr in substrait #8534
- IsNotUnknown feat: implement Unary Expr in substrait #8534
- TryCast
- Sort
- ScalarUDF
- Wildcard
- Exists
- GetIndexedField
- InSubquery Substrait support of InSubquery expressions #8362
- ScalarSubquery
- Alias Substrait doesn't support alias in projection #6489
The following exprs I think we don't need to support:
- OuterReferenceColumn
- Placeholder
Types and scalar
- Interval (YearMonth, DayTime, MonthDayNano) feat: add substrait support for Interval types and literals #10646
- Struct Add support for Substrait Struct literals and type #10622
- Duration (Second, Milli, Micro, Nano)
- View (Binary, List, UTF-8, LargeList)
- Map
- REE
- Time32, Time64
- Union
Meet the spec
- Define *.yaml for datafusion-specific extensions
- Provide URL for extension types
- Support compound signature with function names Support consuming Substrait with compound signature function names #10653
Test
- Pair test with external systems Integrate with the substrait integration test #10710
Others
Things that need further verification:
- Case-sensitivity for both consumer and producer
Describe alternatives you've considered
No response
Additional context
Substrait to DataFusion's physical plan is tracked at #5173