Skip to content

Conversation

@blaginin
Copy link
Collaborator

@blaginin blaginin commented Mar 11, 2025

Which issue does this PR close?

Rationale for this change

We currently override source losing some data, see issue for examples

What changes are included in this PR?

Are these changes tested?

Added tests

Are there any user-facing changes?

No

Comment on lines 382 to 384
if matches!(
projection.input.as_ref(),
LogicalPlan::EmptyRelation(_)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to add some comments to explain why we match EmptyRelation.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, added in 28f5301

Copy link
Collaborator Author

@blaginin blaginin Mar 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had to remove this new test because now (even in main) it produces SELECT * FROM (SELECT [1, 2, 3] AS c) CROSS JOIN UNNEST(c). Feels like this should be discussed separately

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this comment -- doesn't this PR add a new test?

Copy link
Collaborator Author

@blaginin blaginin Mar 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry -- before I added one more test, but then I have to remove it due to ef4a79d - I think we can discuss in that issue and work on it further

@blaginin blaginin marked this pull request as ready for review March 17, 2025 17:58
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this comment -- doesn't this PR add a new test?

if let LogicalPlan::Unnest(unnest) = &p.input.as_ref() {
return self
.unnest_to_table_factor_sql(unnest, query, select, relation);
if let LogicalPlan::Projection(projection) = unnest.input.as_ref()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like something that is a limitation of unnest_to_table_factor_sql -- would it make sense to put the check in there so the check and the code with the assumption are in the same place?

@alamb alamb marked this pull request as draft March 19, 2025 20:01
@alamb
Copy link
Contributor

alamb commented Mar 19, 2025

Marking as draft as I think this PR is no longer waiting on feedback. Please mark it as ready for review when it is ready for another look

@blaginin blaginin marked this pull request as ready for review March 21, 2025 13:57
@blaginin blaginin requested review from alamb and goldmedal March 21, 2025 13:59
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me -- thanks @blaginin

Comment on lines +636 to +647
TestStatementWithDialect {
sql: "SELECT unnest([1, 2, 3, 4]) from unnest([1, 2, 3]);",
expected: r#"SELECT UNNEST([1, 2, 3, 4]) AS UNNEST(make_array(Int64(1),Int64(2),Int64(3),Int64(4))) FROM UNNEST([1, 2, 3])"#,
parser_dialect: Box::new(GenericDialect {}),
unparser_dialect: Box::new(CustomDialectBuilder::default().with_unnest_as_table_factor(true).build()),
},
TestStatementWithDialect {
sql: "SELECT unnest([1, 2, 3, 4]) from unnest([1, 2, 3]);",
expected: r#"SELECT UNNEST([1, 2, 3, 4]) AS UNNEST(make_array(Int64(1),Int64(2),Int64(3),Int64(4))) FROM UNNEST([1, 2, 3])"#,
parser_dialect: Box::new(GenericDialect {}),
unparser_dialect: Box::new(CustomDialectBuilder::default().with_unnest_as_table_factor(true).build()),
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They look like the same. 🤔 I think we can remove one of them.

@goldmedal goldmedal merged commit d68fca9 into apache:main Mar 24, 2025
27 checks passed
@goldmedal
Copy link
Contributor

Thanks @blaginin and @alamb for reviewing. I'll create a follow-up PR to remove the duplicate test.

qstommyshu pushed a commit to qstommyshu/datafusion that referenced this pull request Mar 27, 2025
* Only unnest source for `EmptyRelation`

* Add a note on new condition

* Remove new test

* Put all unnest assumptions into one function
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sql SQL Planner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Different unnests on plan_to_sql are merged

3 participants