-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Move Pruning into physical-optimizer crate
#13485
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
alamb
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @irenjj -- this looks very nice!
I think we need to clean up some of the dependencies, but it is really close
Thank you!
| datafusion-expr = { workspace = true, default-features = true } | ||
| datafusion-functions-nested = { workspace = true } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please move datafusion-functions-nested to dev-dependencies?
I also think we can avoid the dependency on datafusion-expr by changing
use datafusion_expr::Operator;to
use datafusion_expr_common::operator::Operator;|
|
||
| [dependencies] | ||
| arrow = { workspace = true } | ||
| arrow-array = { workspace = true } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If possible, can you avoid this explicit dependency? Whatever is used in arrow-array should be available in arrow and then we can keep the dependency chain simpler
| //! based on statistics (e.g. Parquet Row Groups) | ||
| //! | ||
| //! [`Expr`]: crate::prelude::Expr | ||
| use std::collections::HashSet; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
datafusion-expr dependency is needed by //! [Expr]: crate::prelude::Expr, should we remove the comment to avoid introducing new dependencies? @alamb
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In other places, we have changed the link to docs.rs directly. So in this case, change the link to
https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.Expr.html
| recursive = { workspace = true } | ||
|
|
||
| [dev-dependencies] | ||
| datafusion-expr = { workspace = true } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
needed in test.
alamb
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great to me -- thank you @irenjj
Which issue does this PR close?
part of #11502.
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?