Adds summary output to CLI instrumented object stores

[BlakeOrth]

Which issue does this PR close?

This does not fully close, but is an incremental building block component for:

• [datafusion-cli] Add a way to see what object store requests are made #17207

The full context of how this code is likely to progress can be seen in the POC for this effort:

• POC: datafusion-cli instrumented object store #17266

Rationale for this change

For particularly large requests, in terms of number of objects in a table or large objects, the number of operations for a query may be quite large. In these cases, understanding the aggregate impact of various object store operations is likely the best way to understand the impact those operations had on a particular query. This PR allows users of an instrumented object store to understand and display basic summary statistics related to the RequestDetails collected during a query.

What changes are included in this PR?

• Adds a RequestSummary type for the instrumented object store to display summary statistics about instrumented requests
• Adds a generic Stats type to track the statistics for the summary
• Adds tests for the new code
• Adds a basic summary output to the user-facing display when profiling is enabled
• Adds docs for new and newly exported public items

Are these changes tested?

Yes. The new functionality has tests implemented, aside from testing the actual display output. The functional output can be seen below:

DataFusion CLI v50.1.0
> \object_store_profiling enabled
ObjectStore Profile mode set to Enabled
> CREATE EXTERNAL TABLE hits
STORED AS PARQUET
LOCATION 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet';
0 row(s) fetched.
Elapsed 0.268 seconds.

Object Store Profiling
Instrumented Object Store: instrument_mode: Enabled, inner: HttpStore
2025-10-13T22:15:50.518465131+00:00 operation=Get duration=0.030742s size=8 range: bytes=174965036-174965043 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-13T22:15:50.549263341+00:00 operation=Get duration=0.033060s size=34322 range: bytes=174930714-174965035 path=hits_compatible/athena_partitioned/hits_1.parquet

Summaries:
Get
count: 2
duration min: 0.030742s
duration max: 0.033060s
duration avg: 0.031901s
size min: 8 B
size max: 34322 B
size avg: 17165 B
size sum: 34330 B

>

Are there any user-facing changes?

Yes? Just like the previous PR this does change the user-facing output, but there's no API breaking changes.

cc @alamb

[alamb]

Looks great to me -- thank you @BlakeOrth

[alamb]

As a nice to have future feature, it would be great to have these values be more human readable -- e.g. in stead of 17446363 B have it be 17.4 MB

[alamb]

Another potential improvement is to reduce the number of lines used for display:

From:

Summaries:
Get
count: 2
duration min: 0.030742s
duration max: 0.033060s
duration avg: 0.031901s
size min: 8 B
size max: 34322 B
size avg: 17165 B
size sum: 34330 B

To something like

Summaries :
GET count: 2
duration(min/max/avg): 0.030742s / 0.033060s / 0.031901s
size(min/max/avg/sum): 8 B / 34322 B / 17165 B / 34330 B

but then again maybe that is even harder to read 🤔

[BlakeOrth]

@alamb Yeah, I'm not sure what the best way to display these is in a concise but readable manner. I think in the original issue for this I had also postulated that perhaps a table output, similar to the table information and query results might work.

+----------+-----------+-----------+-----------+---------+
|  Metric  |    min    |    max    |    avg    |   sum   |
+----------+-----------+-----------+-----------+---------+
| duration | 0.030742s | 0.033060s | 0.031901s | N/A     |
| size     | 8 B       | 34322 B   | 17165 B   | 34330 B |
+----------+-----------+-----------+-----------+---------+

Admittedly I'm probably not the right person to make that decision; UI/UX are not my strong suit.

[alamb]

@alamb Yeah, I'm not sure what the best way to display these is in a concise but readable manner. I think in the original issue for this I had also postulated that perhaps a table output, similar to the table information and query results might work.

+----------+-----------+-----------+-----------+---------+
|  Metric  |    min    |    max    |    avg    |   sum   |
+----------+-----------+-----------+-----------+---------+
| duration | 0.030742s | 0.033060s | 0.031901s | N/A     |
| size     | 8 B       | 34322 B   | 17165 B   | 34330 B |
+----------+-----------+-----------+-----------+---------+

Admittedly I'm probably not the right person to make that decision; UI/UX are not my strong suit.

I made a PR with this

• Improve datafusion-cli object store profiling summary display #18085

It isn't ready for review yet but I'll get it ready tomorrow