Skip to content

Disable parquet_encryption by default in datafusion-sqllogictests #18492

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Nov 6, 2025
Merged

Disable parquet_encryption by default in datafusion-sqllogictests #18492

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6b651a0
[issue_18490] Datafuson default will enable encryption, we should dis…
zhuqi-lucas Nov 5, 2025
96b6469
Merge branch 'main' into issue_18490
zhuqi-lucas Nov 5, 2025
e364d1a
fix
zhuqi-lucas Nov 5, 2025
2edfe62
Address comments
zhuqi-lucas Nov 5, 2025
441e4bb
Merge branch 'main' into issue_18490
zhuqi-lucas Nov 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/extended.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ jobs:
rust-version: stable
- name: Run sqllogictest
run: |
cargo test --features backtrace --profile release-nonlto --test sqllogictests -- --include-sqlite
cargo test --features backtrace,parquet_encryption --profile release-nonlto --test sqllogictests -- --include-sqlite
cargo clean


Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -475,7 +475,7 @@ jobs:
export RUST_MIN_STACK=20971520
export TPCH_DATA=`realpath datafusion/sqllogictest/test_files/tpch/data`
cargo test plan_q --package datafusion-benchmarks --profile ci --features=ci -- --test-threads=1
INCLUDE_TPCH=true cargo test --features backtrace --profile ci --package datafusion-sqllogictest --test sqllogictests
INCLUDE_TPCH=true cargo test --features backtrace,parquet_encryption --profile ci --package datafusion-sqllogictest --test sqllogictests
- name: Verify Working Directory Clean
run: git diff --exit-code

Expand Down
5 changes: 4 additions & 1 deletion datafusion/sqllogictest/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ bigdecimal = { workspace = true }
bytes = { workspace = true, optional = true }
chrono = { workspace = true, optional = true }
clap = { version = "4.5.50", features = ["derive", "env"] }
datafusion = { workspace = true, default-features = true, features = ["avro", "parquet_encryption"] }
datafusion = { workspace = true, default-features = true, features = ["avro"] }
Copy link
Contributor

@alamb alamb Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider enabling the default-features too 🤔

Copy link
Contributor Author

@zhuqi-lucas zhuqi-lucas Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @alamb for review, do you mean the default-features to false? Here it use true already, i think enable default-features for datafusion here is ok for sqllogictest.

Copy link
Contributor

@alamb alamb Nov 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was suggesting turning off the default features.

I think given this says default-features=true, that means that anyone who uses datafusion-sqllogictest will get all default features, and there will not be any way to turn the default features off

Copy link
Contributor Author

@zhuqi-lucas zhuqi-lucas Nov 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point @alamb , i think default features are ok, i just tried to set false here, the sqllogictest will fail in my local test.

And here we remove here for parquet_encryption because it removed from datafusion default, also i think the encryption should be always optional instead of the default.

Or we can set to false, if we can easily find the minimal features here?

Copy link
Contributor

@alamb alamb Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should do anything more on this PR

Or we can set to false, if we can easily find the minimal features here?

Yeah that would probably be the ideal outcome (we could file a ticket to track the idea) I don't think it is particularly high priority

Copy link
Contributor Author

@zhuqi-lucas zhuqi-lucas Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed a ticket to track this, thanks!
#18506

datafusion-spark = { workspace = true, default-features = true }
datafusion-substrait = { workspace = true, default-features = true }
futures = { workspace = true }
Expand Down Expand Up @@ -78,6 +78,9 @@ postgres = [
"testcontainers-modules",
"tokio-postgres",
]
parquet_encryption = [
"datafusion/parquet_encryption",
]

[dev-dependencies]
env_logger = { workspace = true }
Expand Down