feat: Parquet modular encryption #16351

corwinjoy · 2025-06-10T00:31:19Z

Which issue does this PR close?

Closes Support integration with Parquet modular encryption #15216.

What changes are included in this PR?

This PR adds support for encryption in DataFusion’s Parquet implementation. The changes introduce new configuration options for file encryption and decryption properties, update various components (including proto conversion, file reading/writing, and tests), and add an end-to-end encrypted Parquet example.

Are these changes tested?

Tests and examples have been added to demonstrate and test functionality versus Parquet modular encryption. These could use feedback since there may be additional DataFusion usage cases that should be covered.

Are there any user-facing changes?

Additional options have been added to allow encryption/decryption configuration. We are soliciting additional feedback on how to handle key columns in a way that best fits the existing API.

Catalog of changes via copilot

Show a summary per file

File	Description
docs/source/user-guide/configs.md	Added documentation for new encryption and decryption properties.
datafusion/sqllogictest/test_files/information_schema.slt	Updated SQL logic tests to include encryption/decryption settings.
datafusion/proto/src/logical_plan/file_formats.rs	Added default None values for encryption properties in proto.
datafusion/proto-common/src/from_proto/mod.rs	Updated conversion to set encryption properties to None.
datafusion/datasource-parquet/src/source.rs	Passed file decryption properties from table options.
datafusion/datasource-parquet/src/opener.rs	Disabled page index when encryption is enabled.
datafusion/datasource-parquet/src/file_format.rs	Propagated decryption properties in schema and statistics fetching.
datafusion/core/tests/parquet/mod.rs	Added the encryption test module reference.
datafusion/core/tests/parquet/encryption.rs	Added tests for round-trip encryption.
datafusion/core/tests/parquet/custom_reader.rs	Updated custom reader to pass None for decryption properties.
datafusion/core/src/datasource/file_format/parquet.rs	Added extra parameters for decryption in metadata/statistics fetching.
datafusion/core/src/dataframe/parquet.rs	Added roundtrip test for encrypted Parquet files.
datafusion/common/src/file_options/parquet_writer.rs	Updated writer options to support encryption properties.
datafusion/common/Cargo.toml	Added the hex dependency.
datafusion-examples/examples/parquet_encrypted.rs	Introduced an example that reads/writes encrypted Parquet files.
datafusion-examples/README.md	Updated examples list to include the encrypted Parquet demo.
Cargo.toml	Enabled encryption feature for the parquet crate.

…uite, column encryption is broken.

Co-authored-by: Adam Reeve <[email protected]>

…operties to use references.

…nfig fields.

… "." instead of "::"

2. Fixed unused header warning in config.rs. 3. Fix test case in encryption.rs to call conversion to ConfigFileDecryption properties correctly.

Co-authored-by: Copilot <[email protected]>

…_encryption

Add an example to read and write encrypted parquet files.

…ing with filter.

…main Signed-off-by: Corwin Joy <[email protected]>

…xample Signed-off-by: Corwin Joy <[email protected]>

Signed-off-by: Adam Reeve <[email protected]>

datafusion/common/src/config.rs

adamreeve · 2025-06-26T01:00:15Z

datafusion/datasource-parquet/src/file_format.rs

    store: &dyn ObjectStore,
    meta: &ObjectMeta,
    size_hint: Option<usize>,
+    decryption_properties: Option<&FileDecryptionProperties>,


This looks like a breaking change (and same for fetch_statistics). Should we introduce a new method with this parameter added to avoid breaking the API? It is documented as "subject to change" though so maybe this is ok...

I guess this probably needs to change anyway to handle having an encryption feature.

adamreeve · 2025-06-26T01:03:52Z

datafusion/proto-common/src/from_proto/mod.rs

                .unwrap(),
            column_specific_options,
            key_value_metadata: Default::default(),
+            crypto: Default::default(),


If we aren't going to support conversion to protobuf yet, we should probably raise an error in impl TryFrom<&TableParquetOptions> for protobuf::TableParquetOptions if any crypto options are set.

Should we? I see that key_value_metadata is always set to empty here despite being defined in datafusion/proto-common/src/generated/prost.rs. So, it's not clear to me that we should throw an error here.

Use ConfigFileDecryptionProperties in ParquetReadOptions Signed-off-by: Corwin Joy <[email protected]>

Signed-off-by: Corwin Joy <[email protected]>

alamb · 2025-06-27T20:27:19Z

I merged up to resolve a conflict

alamb · 2025-06-27T21:02:27Z

datafusion/common/src/config.rs

+    /// See ConfigFileEncryptionProperties and ConfigFileDecryptionProperties in datafusion/common/src/config.rs
+    /// These can be set via 'format.crypto', for example:
+    /// ```sql
+    /// OPTIONS (


alamb

Thank you @corwinjoy -- this looks wonderful to me

alamb · 2025-06-27T21:40:39Z

https://github.com/apache/datafusion/actions/runs/15936215612/job/44956521967?pr=16351

That is a new failure for me

     Running bin/sqllogictests.rs (target/ci/deps/sqllogictests-19519efa72c14141)
External error: 1 errors in file /Users/runner/work/datafusion/datafusion/datafusion/sqllogictest/test_files/union_by_name.slt

1. query is expected to fail, but actually succeed:
[SQL] select x, y, z from t3 union all by name select z, y, x, 'd' as zz from t3;
at /Users/runner/work/datafusion/datafusion/datafusion/sqllogictest/test_files/union_by_name.slt:343

I'll trigger it again and see if I can reproduce

alamb · 2025-06-28T13:48:00Z

Thanks again @corwinjoy / @adamreeve and everyone else. This is great

corwinjoy · 2025-06-30T22:55:56Z

Thanks @alamb much appreciated for the review and helpful feedback! We hope to have a followup PR soon with a config to make encryption optional.

alamb · 2025-07-01T18:55:12Z

Thank you @corwinjoy

corwinjoy and others added 30 commits May 29, 2025 19:04

Initial commit to form PR for datafusion encryption support

e668b99

Add tests for encryption configuration

d38dba4

Apply cargo fmt

5a2b456

Add a roundtrip encryption test to the parquet tests.

c972676

cargo fmt

ec3f828

Update test to add decryption parameter to called functions.

3538a27

Try to get DataFrame.write_parquet to work with encryption. Doesn't q…

a754992

…uite, column encryption is broken.

Update datafusion/datasource-parquet/src/opener.rs

e430672

Co-authored-by: Adam Reeve <[email protected]>

Update datafusion/datasource-parquet/src/source.rs

7fcba70

Co-authored-by: Adam Reeve <[email protected]>

Fix write test in parquet.rs

d6b1fca

Simplify encryption test. Remove unused imports.

3353186

Run cargo fmt.

e4bc0e3

Further streamline roundtrip test.

f52e79c

Change From methods for FileEncryptionProperties and FileDecryptionPr…

5615ac8

…operties to use references.

Change encryption config to directly hold column keys using custom co…

61bc78e

…nfig fields.

Fix generated field names in visit for encryptor and decryptor to use…

a81855f

… "." instead of "::"

1. Disable parallel writes with enccryption.

4cf12b3

2. Fixed unused header warning in config.rs. 3. Fix test case in encryption.rs to call conversion to ConfigFileDecryption properties correctly.

cargo fmt

f29bec3

Update datafusion/common/src/file_options/parquet_writer.rs

86fe04b

Co-authored-by: Copilot <[email protected]>

fix variables shown in information schema test.

d4ea63f

Merge remote-tracking branch 'origin/parquet_encryption' into parquet…

0fcc4a5

…_encryption

Backout bad suggestion from copilot

86db3a5

Remove unused serde reference

b34441a

Add an example to read and write encrypted parquet files.

cargo fmt

668d728

change file_format.rs to use global encryption options in struct.

ec1e8da

Turn off page_index for encrypted example. Get encrypted example work…

e233408

…ing with filter.

Tidy up example output.

9ffaae4

Add missing license. Run taplo format

8e244e9

Update configs.md by running dev/update_config_docs.sh

2871d51

Cargo fmt + clippy changes.

c405167

corwinjoy force-pushed the parquet_encryption branch from a3f0721 to 6a77842 Compare June 25, 2025 22:12

corwinjoy added 2 commits June 25, 2025 15:44

Update code to add missing ParquetOpener parameter due to merge from …

9972ecb

…main Signed-off-by: Corwin Joy <[email protected]>

Add CLI documentation for Parquet options and provide an encryption e…

068e65d

…xample Signed-off-by: Corwin Joy <[email protected]>

github-actions bot added the documentation Improvements or additions to documentation label Jun 26, 2025

Use ConfigFileDecryptionProperties in ParquetReadOptions

ecd5f93

Signed-off-by: Adam Reeve <[email protected]>

adamreeve reviewed Jun 26, 2025

View reviewed changes

corwinjoy added 3 commits June 25, 2025 21:07

Merge pull request #6 from adamreeve/parquet_encryption_fix

d79fdf2

Use ConfigFileDecryptionProperties in ParquetReadOptions Signed-off-by: Corwin Joy <[email protected]>

Implement default for ConfigFileEncryptionProperties

f3e6945

Signed-off-by: Corwin Joy <[email protected]>

Add sqllogictest for parquet with encryption

a7005a3

Signed-off-by: Corwin Joy <[email protected]>

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Jun 26, 2025

corwinjoy and others added 2 commits June 26, 2025 15:52

Apply prettier changes from CI

48166f4

Signed-off-by: Corwin Joy <[email protected]>

Merge remote-tracking branch 'apache/main' into parquet_encryption

742a485

alamb reviewed Jun 27, 2025

View reviewed changes

logical conflift

f49f1e8

alamb approved these changes Jun 27, 2025

View reviewed changes

alamb added 2 commits June 27, 2025 17:20

Merge remote-tracking branch 'apache/main' into parquet_encryption

7f4706a

fix another logical conflict

861b54a

Merge branch 'main' into parquet_encryption

ca059b2

alamb merged commit d66d6b9 into apache:main Jun 28, 2025
30 checks passed

This was referenced Jul 2, 2025

feat: Add a configuration to make parquet encryption optional #16649

Merged

Add a feature to make encryption optional #16650

Open

This was referenced Jul 10, 2025

Support multi-threaded writing of encrypted Parquet files #16737

Closed

feat: support multi-threaded writing of Parquet files with modular encryption #16738

Merged

alamb mentioned this pull request Jul 12, 2025

Release DataFusion 49.0.0 (July 2025) #16235

Closed

34 tasks

adamreeve mentioned this pull request Jul 15, 2025

More flexible Parquet encryption configuration #16778

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Parquet modular encryption #16351

feat: Parquet modular encryption #16351

Uh oh!

corwinjoy commented Jun 10, 2025 •

edited

Loading

Uh oh!

Uh oh!

adamreeve Jun 26, 2025

Uh oh!

adamreeve Jun 26, 2025

Uh oh!

adamreeve Jun 26, 2025

Uh oh!

corwinjoy Jun 26, 2025

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

alamb Jun 27, 2025

Uh oh!

alamb left a comment

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

Uh oh!

alamb commented Jun 28, 2025

Uh oh!

corwinjoy commented Jun 30, 2025

Uh oh!

alamb commented Jul 1, 2025

Uh oh!

Uh oh!

feat: Parquet modular encryption #16351

feat: Parquet modular encryption #16351

Uh oh!

Conversation

corwinjoy commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Catalog of changes via copilot

Uh oh!

Uh oh!

adamreeve Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

adamreeve Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

adamreeve Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

corwinjoy Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

alamb Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

Uh oh!

alamb commented Jun 28, 2025

Uh oh!

corwinjoy commented Jun 30, 2025

Uh oh!

alamb commented Jul 1, 2025

Uh oh!

Uh oh!

corwinjoy commented Jun 10, 2025 •

edited

Loading