Support datafusion-cli access to public S3 buckets that do not require authentication

[alamb]

Which issue does this PR close?

• Closes Support datafusion-cli access to public S3 buckets that do not require authentication #16299
• Related to [DISCUSSION] Make it easier and faster to query remote files (S3, iceberg, etc)  #13456

Rationale for this change

I want to be able to access public s3 buckets without providing (valid) s3 credentials

What changes are included in this PR?

1. Add skip_signature option to datafusion-cli CREATE EXTERNAL TABLE
2. Default to skip_signature when other credentials are not provided
3. Update documentation

Before this PR:

DataFusion CLI v47.0.0
> CREATE EXTERNAL TABLE nyc_taxi_rides
STORED AS PARQUET LOCATION 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/';
Object Store error: Generic S3 error: the credential provider was not enabled

After this PR:

DataFusion CLI v48.0.0
> CREATE EXTERNAL TABLE nyc_taxi_rides
STORED AS PARQUET LOCATION 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/';
selec0 row(s) fetched.
Elapsed 1.575 seconds.

> select count(*) from nyc_taxi_rides;
+------------+
| count(*)   |
+------------+
| 1310903963 |
+------------+
1 row(s) fetched.
Elapsed 3.011 seconds.

Are these changes tested?

Yes, new unit tests are added and I tested it manually

For example, if you provide credentials, they take precidence over the signature:

AWS_ACCESS_KEY_ID=foo AWS_SECRET_ACCESS_KEY=bar  cargo run -p datafusion-cli

> CREATE EXTERNAL TABLE nyc_taxi_rides
STORED AS PARQUET LOCATION 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/';
Object Store error: Generic S3 error: Error performing list request: Error performing GET https://s3.us-east-1.amazonaws.com/altinity-clickhouse-data?list-type=2&prefix=nyc_taxi_rides%2Fdata%2Ftripdata_parquet%2F in 134.200375ms - Server returned non-2xx status code: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>foo</AWSAccessKeyId><RequestId>ZAEM63Q02FQXYMTA</RequestId><HostId>mYh2PUtKzDxjrPA4vQm4d+Qae9TiNpCUDDTS5BP4jTayKVE4BQbSpT/+HSIAdzt3lne6G0sxNmE=</HostId></Error>

But you can override this with SKIP_SIGNATURE

> CREATE EXTERNAL TABLE nyc_taxi_rides
STORED AS PARQUET LOCATION 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/' OPTIONS(AWS.SKIP_SIGNATURE 'true');
0 row(s) fetched.
Elapsed 1.455 seconds.

Are there any user-facing changes?

Easier to use datafusion-cli

[alamb]

this changes the default so we don't try and create a signature if no credentials can be extracted

[alamb]

here is support for passing through skip_signature

[alamb]

Looking at this with whitespace blind diff https://github.com/apache/datafusion/pull/16300/files?w=1

makes it easier to see what is happening

Specifically, I just refactored the parsing / table_options extraction into a separate function to reduce repetition.

Otherwise the existing tests are the same

[blaginin]

🙌🏻

[blaginin]

nit: does it make sense to expect specific errors? For example, for CredentialsError::InvalidConfiguration or InvalidConfiguration::ProviderTimedOut, we probably still want to raise since those mean the creds could be set?

[alamb]

That is a great idea -- fixed in 37d1132

[alamb]

Thanks @blaginin !

[alamb]

There appears to be some problem with this code in PR:

• CI failure on Datafusion extended tests / cargo test hash collisions (amd64) (push) #16378
  I have another PR up to add some better error messages to debug:
• Add more context to error message for datafusion-cli config failure #16379