[EPIC] More Subquery support

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

DataFusion supports many types of subqueries (originally tracked via #2248). This epic tracks additional features / support

See also the original design document.

• Implement physical plan for IN Subquery #1835
• Implement physical plan for EXISTS subquery #123
• [Feature] (All) logical optimizers should optimize subqueries #2480
• General framework to decorrelate the subqueries #5492
• Additional ideas for optimizing subqueries #5484
• Implement physical execution of uncorrelated scalar subqueries #3781
• Add support for wider range of scalar subqueries #3725
• Add support for scalar subqueries in BETWEEN expressions #3667
• scalar_subquery_to_join internal error #3417
• Validate subquery expressions #2361
• Add support for correlated subquery #6140
• decorrelate_where_in reports error when optimizing limit subquery #5808
• SQL case, This feature is not implemented: Physical plan does not support logical expression EXISTS (<subquery>) #5789
• Error when using DISTINCT in a filter clause using a subquery #5633
• Decorrelate scalar subqueries with more complex filter expressions #14554
• Nested correlated subquery error with a depth exceeding 1 #15558

[Y-jiji]

I want to try this for GSoC. Leave an email: tyang425@gatech.edu

[yashdavkhar7020]

🚀 Interested in Contributing to Correlated Subqueries in DataFusion

Hi

I am Yash Davkhar, and I am highly interested in contributing to this project on adding support for correlated subqueries in DataFusion. I have experience in SQL, database systems, and query optimization and would love to apply my skills to improve DataFusion’s SQL engine.

I have gone through the project description and the linked epic, and I would appreciate any guidance on how to get started. Could you please share any specific tasks or areas I should focus on first?

Looking forward to contributing! 🚀

Best regards,
Yash Davkhar

[SanjayUG]

Hi there,

Looks interesting issue, I would love to contribute to it without any fiction.

Thank You.

[PhaniKulkarni]

Hi, I am interested in contributing to Apache DataFusion, specifically in implementing correlated subqueries support

I have experience in SQL Engines, Data Migration and query optimization and have gone through DataFusion’s documentation and source code. However, I would appreciate some guidance on:

1. Where to start in understanding DataFusion’s query planning and optimization workflow.

2. Any specific resources or previous discussions that would be helpful.

3. How best to break down the implementation steps before submitting a PR.

A brief discussion would be greatly appreciated.

Here is my contact and profile details:
LinkedIn: https://www.linkedin.com/in/phani-kulkarni-140674180/
Mail: kulkarniphani@gmail.com

Thanks and Regards,
Phani Kulkarni

[alamb]

Where to start in understanding DataFusion’s query planning and optimization workflow.

I think starting at https://datafusion.apache.org/ and https://datafusion.apache.org/contributor-guide/architecture.html would probably be a good place to start

[alamb]

• I am trying to organize a join task force for planning joins / subqueries: [DISCUSSION] JOIN "task force" / project team #15885