Hello from fsspec!

[martindurant]

I see that we have some goals in common. fsspec is a mature library in use by Dask, pandas, zarr and others. However, we don't have a good implementation of pathlib on our API. Would you like to join forces to implement pathlib-on-fsspec, using the code you have already developed?

xref: fsspec/filesystem_spec#434

another attempt to do this, explicitly for fsspec: https://github.com/datarevenue-berlin/drfs

[pjbull]

👋 @martindurant sounds cool!

There are probably a few level of integration that could work well. We intend to make it pretty easy to add custom clients/backends by implementing a relatively small surface area. We're adding more docs on this, but here's the boto3 one as an example. Implementing each of the functions there should make it simple to enable all of the pathlib API that we support:

https://github.com/drivendataorg/cloudpathlib/blob/8692c596d98292c4ca44fd2e654effb595db1427/cloudpathlib/s3/s3client.py

Option (1):
I could see a FsspecClient as part of cloudpathlib that if you pip install cloudpathlib[fsspec] then it uses the fsspec client for any of the schemes that fsspec supports. There's probably some details in the coupling between the specific kind of path object we return (e.g., S3Path) and the Client object that would need to be worked out, but that all seems pretty doable.

Option (2):
You could that same extension work for a custom Client and CloudPath object within fsspec and the expose the factory methods as you see fit in your API.

Other thoughts on interoperability? There's definitely a big chunk of cool things in fsspec that we don't want to duplicate (chunks/buffering, compressing cache, async, etc.).

[martindurant]

I would say those two options sound pretty similar, except for where the code lives - which I am not too worried about at all. You are probably more likely to get to it, if you have time to experiment - but seeing that this was already requested at fsspec, maybe someone else in the community contributes some pathlib interface anyway.

Do you think there's an argument for S3FileSystem, GCSFileSystem and FTPFileSystem from fsspec-land to supplant the implementations you have or are developing? They may be more complete and reduce the amount of work you need to do.

[jayqi]

@martindurant @pjbull

I took a quick stab at making FsspecClient and FsspecPath classes that is backed by an fsspec FileSystem in #109.

from cloudpathlib.fsspec import FsspecClient, FsspecPath
from s3fs import S3FileSystem

FsspecClient(filesystem=S3FileSystem()).set_as_default_client()

cp = FsspecPath("s3://drivendata-public-assets/")
list(cp.iterdir())[0:3]
#> [FsspecPath('s3://drivendata-public-assets/CIP-from-brochure.jpg'), FsspecPath('s3://drivendata-public-assets/EPD-report.png'), FsspecPath('s3://drivendata-public-assets/Ending-Extreme-Poverty_A-Focus-on-Children.png')]

I think the tricky part that I don't have a great sense of is how to design the right interface to let users choose to use fsspec and to initialize the correct fsspec FileSystem class with minimal friction. In cloudpathlib currently, we already have a way of specifying dependencies with extras_require and with class dispatching based on the prefix/protocol in the bath. Choosing to use fsspec and dispatching to the right fsspec filesystem, and having dependencies specified is like two more layers of user choices.

The question of moving our backend logic entirely to fsspec is a pretty big one and would involve a lot of changes. This is my first time using fsspec, so I still don't have a great feel for the fsspec design or ecosystem.

[martindurant]

Typical usage in fsspec allows access to the set of backend implementations using a protocol string and arguments, e.g., fsspec.filesystem("s3", anon=True), or by passing URLs including a prefix fsspec.open("s3://bucket/path/file", anon=True). (anon is just an example here, specific to s3)

Many packages that use fsspec, and some internal logic too, uses storage_options as a dict of kwargs to pass to the main entry functions, e.g.,
pd.read_csv("s3://bucket/path/file", storage_options={"anon": True}).

The current set of optional deps for fsspec are:

    extras_require={
        "abfs": ["adlfs"],
        "adl": ["adlfs"],
        "dask": ["dask", "distributed"],
        "dropbox": ["dropboxdrivefs", "requests", "dropbox"],
        "gcs": ["gcsfs"],
        "git": ["pygit2"],
        "github": ["requests"],
        "gs": ["gcsfs"],
        "hdfs": ["pyarrow"],
        "http": ["requests", "aiohttp"],
        "sftp": ["paramiko"],
        "s3": ["s3fs"],
        "smb": ["smbprotocol"],
        "ssh": ["paramiko"],
    },

but we don't depend on anything just to import the package, and some of the backends like file and memory. Sounds like we came up with a similar set of solutions!

Note that there are some implementations that are meant/able to be used with a further target filesystem, such as the local caches.

I'd be happy to talk about how fsspec works and describe our design/ecosystem.

[remi-braun]

Hello there !

I would like to point out that the support of fsspec would be incredibly useful for me, as for now I need to cache my pandas and geopandas files before trying to open them.

Thanks a lot :)

[pjbull]

@remi-braun Thanks for the vote for fsspec! So that I understand your use case, is the reason that you want an fsspec implementation that you want to be able to do streaming read/write rather than downloading the whole file to a local cache first?

FYI, @jayqi did a good implementation of fsspec as a backend in #109. However, it's different from just a pathlib API for fsspec since it uses the cloudpathlib caching logic and business logic by implementing our *Client and *Path classes. To really take advantage of their approach to caching and reading/writing we'd likely need to re-implement a lot of the reading/writing (and even things like listing which can be cached in fsspec).

(Here's the issue just on streaming: #9 )

[remi-braun]

Arf sadly it is much more prosaic:
my prime concern is to unify my code using agnostically files in the cloud or on disk and for now it is not possible :

path = AnyPath(path)

# Load vector in cache if needed (geopandas and cloudpathlib are not compatible for now)
if isinstance(path, CloudPath):
    path = AnyPath(path.fspath)

vect = geopandas.read_file(vect_path)

However I do not use files big enough to see any slowdown caching files in my usecase (apart from rasterio rasters, but they do not use fsspec if I'm correct)

[pjbull]

Ah, interesting. That looks to me like a geopandas feature request since they don't support os.PathLike. They do say they support filelike objects. You could try the following and see if it works. If they support filelike objects then I think it should be fine with CloudPath or Path objects:

path = AnyPath(path)

with path.open() as f:
    vect = geopandas.read_file(f)

Recent versions of pandas should work fine with CloudPath. For example:

In [1]: import pandas as pd

In [2]: from cloudpathlib import CloudPath

In [3]: df = pd.read_csv(CloudPath('s3://drivendata-public-assets/odsc-west-2019/california-tracts.csv'))

In [4]: df.head()
Out[4]:
        GEOID  year    name             parent-location  population  ...  eviction-rate  eviction-filing-rate  low-flag  imputed  subbed
0  6001400100  2000  4001.0  Alameda County, California     2497.87  ...           0.00                  0.00         1        0       0
1  6001400100  2001  4001.0  Alameda County, California     2497.87  ...           0.00                  0.86         1        0       0
2  6001400100  2002  4001.0  Alameda County, California     2497.87  ...           0.80                  0.80         1        0       0
3  6001400100  2003  4001.0  Alameda County, California     2497.87  ...           1.49                  1.49         1        0       0
4  6001400100  2004  4001.0  Alameda County, California     2497.87  ...           0.00                  0.00         1        0       0

[5 rows x 27 columns]

[martindurant]

I think the point of the discussion is that we could use cloudpath to have fsspec support generic pathlike objects without having to ask third-party libraries such as geopandas to do extra work.

[remi-braun]

Ah, interesting. That looks to me like a geopandas feature request since they don't support os.PathLike. They do say they support filelike objects. You could try the following and see if it works. If they support filelike objects then I think it should be fine with CloudPath or Path objects:

path = AnyPath(path)

with path.open() as f:
    vect = geopandas.read_file(f)

Recent versions of pandas should work fine with CloudPath. For example:

In [1]: import pandas as pd

In [2]: from cloudpathlib import CloudPath

In [3]: df = pd.read_csv(CloudPath('s3://drivendata-public-assets/odsc-west-2019/california-tracts.csv'))

In [4]: df.head()
Out[4]:
        GEOID  year    name             parent-location  population  ...  eviction-rate  eviction-filing-rate  low-flag  imputed  subbed
0  6001400100  2000  4001.0  Alameda County, California     2497.87  ...           0.00                  0.00         1        0       0
1  6001400100  2001  4001.0  Alameda County, California     2497.87  ...           0.00                  0.86         1        0       0
2  6001400100  2002  4001.0  Alameda County, California     2497.87  ...           0.80                  0.80         1        0       0
3  6001400100  2003  4001.0  Alameda County, California     2497.87  ...           1.49                  1.49         1        0       0
4  6001400100  2004  4001.0  Alameda County, California     2497.87  ...           0.00                  0.00         1        0       0

[5 rows x 27 columns]

Sadly it is not the case, and I don't know why...
I haven't investigated though, but my workaround is sufficient for now 😉

[pjbull]

Thanks, @martindurant. How would you use fsspec when opening a file with geopandas? I'm curious what fsspec implements that would make this work without having to explicitly download to a local file first and point to that local path.

(That said, I would encourage library maintainers to consider supporting os.PathLike since it is the official Python way to try to go from an object to a local file path.)

(Also, would be curious about your thoughts on testing #109 to see if it is viable! The fact it doesn't take advantage of all of the hard work you've done on caching, streaming, etc. does make it seem less appealing.)

[martindurant]

what fsspec implements that would make this work without having to explicitly download to a local file first and point to that local path

fsspec can do this job implicitly; but yes, there are libraries which, typically because of some inner C code, can only work with real OS file handles on the local filesystem.

it is the official Python way to try to go from an object to a local file path

This is an exaggeration! :)

On #109, I think it's totally the way forward. For configuring the relevant backends, you could rely on the fsspec config system, or maybe allow extra kwargs when instantiating a Path, which get passed to other downstream Paths.