ObjectStore Directory Semantics

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

LocalFileSystem interprets the prefix passed to ObjectStore::list_file as the path to a directory, and then proceeds to enumerate this directory recursively. S3FileSystem, however, interprets the prefix as a string prefix.

The distinction arises if you consider a file structure like

foo/a.txt
foo/b.txt

If called with a prefix of fo, LocalFileSystem will return an error, whereas S3FileSystem will return both files.

Describe the solution you'd like

I personally would expect something called ObjectStore to behave like an object store, and not a filesystem. In particular I would expect it to behave like a KV store without any notion of directories.

I would therefore suggest:

• Remove FileSystem from the naming of the implementations
• Map object storage semantics to the local filesystem, as opposed to mapping filesystem semantics to all object storages

Describe alternatives you've considered

We could instead call the trait something like FileSystem and give it file system like semantics.

Additional context

I noticed this whilst reviewing #2394 - it seems off to me that we should need to split based on path delimiters given object stores don't have such a concept.

Thoughts @matthewmturner @alamb @timvw ?

[timvw]

Regardless the chosen approach (ObjectStore vs FileSystem) I would consider to make the trait (and it's methods) consistent:

Currently the trait is named ObjectStore but it only has methods related to Files. Either update/rename the methods (and datatypes) such as fn list_object(s) -> ObjectMetadata .. Or rename the trait to FileSystem...

[timvw]

Also consider Issue-2465.

The objectstore is requested to list files that match prefix "/Users/blah//".
LocalFileSystem returns items such as "/Users/blah/test.txt" .

One could claim that this path does not match the prefix.
One could also claim that an object can have multiple keys and that this file has an alternative key /Users/blah//test.txt" which does match the prefix.

For globbing to work, within a key/prefix concept, the returned objects/files should carry the "key" that matches the prefix. (Currently my fix in the mentioned issue works the other way round -> globbing is adapted to filesystem implementation).

[wjones127]

Also relevant:

• Add write and delete operations to ObjectStore #2246
• ObjectStore write support #2185

As part of that PR, I plan on creating a generic suite of tests to validate a ObjectStore implementation, and that could enforce these behavior expectations for each implementation.

For FileSystem vs ObjectStore, I'm only familiar with implementations of the first in the context of query engines (such as Arrow C++'s FileSystem or Python's fsspec). Are there examples of ObjectStore implementations?

My preference is for a "FileSystem" approach since that's more familiar, but open the ObjectStore approach as long as that can be used to read and write in a way compatible with other systems that may use a FileSystem approach. (For example, the current implementation doesn't force delimiting paths with /, but I think that expected by other systems.)

[tustvold]

Are there examples of ObjectStore implementations

I'm not sure what you mean by this, but object stores are really just key value stores with a vaguely RESTful API, i.e.

• PutObject - associate an object (set of bytes) with a string key, replacing any existing value
• GetObject - get the object associated with a key
• CopyObject - copy the object associated with one key, to another
• ListObjects - list the keys with a given prefix
• DeleteObject - delete the value with a given key

There are more complex APIs for things like multipart uploads, bucket creation, etc... but in terms of what a client would be interested in that is the entirety of the API. To put it another way, the interface of object storage is significantly less expressive than that of a filesystem. This is why object storage is scalable, and things like NFS, EFS, are... not 😅

Trying to make object storage behave exactly like a filesystem is impossible (e.g. S3 doesn't support CreateIfNotExists), however, my thesis is that no query engine actually wants filesystem semantics, and this is why these linked abstractions kind of work (#2205 (comment)).

My suggestion is that by instead implementing the less expressive object storage semantics, we can avoid a whole host of funky edge-cases around directories, paths, buffering, read-ahead etc...

in a way compatible with other systems that may use a FileSystem approach

Could you expand on what you mean by this, do you mean being able to read data written by another system which should be trivial, or are you talking about some sort of API-level integration like FFI?

[alamb]

Are there examples of ObjectStore implementations?

The canonical example of ObjectStore is AWS's S3: https://aws.amazon.com/s3/ and then there are many distributed storage systems that present a similar interface, as @tustvold describes in #2445 (comment)

The idea of the "ObjectStore" interface in DataFusion was to provide API access to the lowest common denominator feature set across several storage implementations. For example, here are three implementations for S3, HDFS, and Azure specifically:

• https://github.com/datafusion-contrib/datafusion-objectstore-s3
• https://github.com/datafusion-contrib/datafusion-objectstore-hdfs
• https://github.com/datafusion-contrib/datafusion-objectstore-azure

In terms of "glob"ing, that is typically not a feature provided by object stores (e.g. there is no such thing in S3, which instead offers a much more restricted notion of prefixes). Thus, it seems to me if we want to support globbing for DataFusion when running on local files, it will have to be a special case somehow.

You can see another example of a Rust API to object storage in IOx: https://github.com/influxdata/influxdb_iox/blob/main/object_store

[alamb]

It would help me significantly, to understand the globbing usecase more -- like when exactly are you selecting a subset of files in a directory via a glob? Most analytic systems I have seen tend to assume data has been pre-grouped into directories (or equivalent)

AWS redshift does offer the ability to specify a subset of files that are not all in the same directory, but it does so by taking a manifest file: https://docs.aws.amazon.com/redshift/latest/dg/loading-data-files-using-manifest.html

[alamb]

Also, @carols10cents spent considerable time sorting out consistent directory semantics for object stores and local files in https://github.com/influxdata/influxdb_iox/blob/main/object_store -- maybe we can just use those semantics (or maybe even the code?)

[wjones127]

I'm not sure what you mean by this

Sorry that wasn't clear. I pointed out two implementations of an abstraction over object stores (S3, GCS, etc.) that are like filesystems (in that they have a notion of directories, not that they make any guarantees about atomicity). These are used by analytics systems like Dask and PyArrow, so there's some evidence we can build useful query engines on top of such an abstraction.

Thanks @alamb for the IOx example.

Trying to make object storage behave exactly like a filesystem is impossible (e.g. S3 doesn't support CreateIfNotExists), however, my thesis is that no query engine actually wants filesystem semantics,

I largely agree. I think the main thing these "FileSystem" abstractions provide is a notion of "directory", which is important in directory-partitioned datasets. The existing API can handle that fine with delimiter, but it does seem a little funny you can provide whatever delimiter you want.

Could you expand on what you mean by this, do you mean being able to read data written by another system which should be trivial, or are you talking about some sort of API-level integration like FFI?

Yeah I think as long as you could do the expected filesystem operations on top of the API, then that seems fine. For context, I plan to wrap the ObjectStore API in a PyArrow-compatible filesystem for use in delta-rs. Hence #2246.

But I think I'll scale back my changes in #2246 and remove the create_dir(), remove_dir() methods if we want to just think of this as an object store abstraction with no awareness of directories.

Also, @carols10cents spent considerable time sorting out consistent directory semantics for object stores and local files in https://github.com/influxdata/influxdb_iox/blob/main/object_store -- maybe we can just use those semantics (or maybe even the code?)

That sounds very promising @lamb. Thanks for pointing out!

[tustvold]

If we like the IOx object store interface and want to reuse the implementation, I can probably see about getting it published to crates.io, just let me know. It wasn't my intent with this issue, rather I just wanted clarity on what I should be reviewing 😅, but I would be happy to help make it happen if there is consensus on it being a good idea

[wjones127]

If we like the IOx object store interface and want to reuse the implementation, I can probably see about getting it published to crates.io, just let me know.

I would be supportive of that, but we probably would need to discuss what that means for
https://github.com/datafusion-contrib/datafusion-objectstore-s3
https://github.com/datafusion-contrib/datafusion-objectstore-hdfs
https://github.com/datafusion-contrib/datafusion-objectstore-azure

Do we want to create a new issue to discuss that?

[timvw]

@alamb The globbing is mainly relevant in raw/ingestion folders...

Eg: we have end up with a structure such as:
/nyc-taxidata/input/yellow_tripdata_2021-11.csv
/nyc-taxidata/input/yellow_tripdata_2021-12.csv
/nyc-taxidata/input/yellow_tripdata_2022-01.csv
/nyc-taxidata/input/green_tripdata_2021-12.csv
/nyc-taxidata/input/green_tripdata_2022-01.csv
/nyc-taxidata/input/green_tripdata_2022-02.csv

In a typical job we would then process and prepare the data for consumption:
/nyc-taxidata/accepted/yellow_tripdata/year=2022/month=1/blah.parquet
/nyc-taxidata/accepted/green_tripdata/year=2022/month=1/blah.parquet

I don't need access to all sorts of key filters (compared to all key filters in a system such as HBase but globbing is not something I would push back to the end-user (In hadoop this is also supported by alternative (s3, azure) hadoop filesystem implementations)

[timvw]

In summary, I agree with the ObjectStore semantics being sufficient.

I also do want to point out that globbing is nothing more than making the suffix filter more powerful (instead of matching against a static suffix (eg: ".parquet") it allows matching against a pattern).

/// Calls `list_file` with a suffix filter
async fn list_file_with_suffix(
    &self,
    prefix: &str,
    suffix: &str,
) -> Result<FileMetaStream>