You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
or optionally, ifAZURE_STORAGE_ACCOUNT_NAMEand an AZURE_STORAGE_<CREDENTIAL>is
@@ -58,15 +48,9 @@ ddf = dd.read_parquet('az://nyctlc/green/puYear=2019/puMonth=*/*.parquet', stora
58
48
59
49
Details
60
50
-------
61
-
The package includes pythonic filesystem implementations for both
62
-
Azure Datalake Gen1 and Azure Datalake Gen2, that facilitate
63
-
interactions between both Azure Datalake implementations and Dask. This is done leveraging the
64
-
[intake/filesystem_spec](https://github.com/intake/filesystem_spec/tree/master/fsspec) base class and Azure Python SDKs.
51
+
The package includes pythonic filesystem implementations for both [Azure Blobs](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-overview) and [Azure Datalake Gen2 (ADLS)](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction), that facilitate interactions between these implementations and Dask. This is done leveraging the [fsspec/filesystem_spec](https://github.com/fsspec/filesystem_spec) base class and Azure Python SDKs.
65
52
66
-
Operations against both Gen1 Datalake currently only work with an Azure ServicePrincipal
67
-
with suitable credentials to perform operations on the resources of choice.
68
-
69
-
Operations against the Gen2 Datalake are implemented by leveraging [Azure Blob Storage Python SDK](https://github.com/Azure/azure-sdk-for-python).
53
+
Operations against Azure Blobs and ADLS Gen2 are implemented by leveraging [Azure Blob Storage Python SDK](https://github.com/Azure/azure-sdk-for-python).
70
54
71
55
### Setting credentials
72
56
The `storage_options` can be instantiated with a variety of keyword arguments depending on the filesystem. The most commonly used arguments are:
@@ -81,7 +65,7 @@ The `storage_options` can be instantiated with a variety of keyword arguments de
81
65
anonymous access will not be attempted. Otherwise the value for `anon` resolves to True.
82
66
-`location_mode`: valid values are "primary" or "secondary" and apply to RA-GRS accounts
83
67
84
-
For more argument details see all arguments for [`AzureBlobFileSystem` here](https://github.com/fsspec/adlfs/blob/f15c37a43afd87a04f01b61cd90294dd57181e1d/adlfs/spec.py#L328) and [`AzureDatalakeFileSystem` here](https://github.com/fsspec/adlfs/blob/f15c37a43afd87a04f01b61cd90294dd57181e1d/adlfs/spec.py#L69).
68
+
For more argument details see all arguments for [`AzureBlobFileSystem` here](https://fsspec.github.io/adlfs/api/#adlfs.AzureBlobFileSystem)
85
69
86
70
The following environmental variables can also be set and picked up for authentication:
87
71
- "AZURE_STORAGE_CONNECTION_STRING"
@@ -102,3 +86,6 @@ The filesystem can be instantiated for different use cases based on a variety of
102
86
The `AzureBlobFileSystem` accepts [all of the Async BlobServiceClient arguments](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python).
103
87
104
88
By default, write operations create BlockBlobs in Azure, which, once written can not be appended. It is possible to create an AppendBlob using `mode="ab"` when creating and operating on blobs. Currently, AppendBlobs are not available if hierarchical namespaces are enabled.
89
+
90
+
### Older versions
91
+
ADLS Gen1 filesystem has officially been [retired](https://learn.microsoft.com/en-us/lifecycle/products/azure-data-lake-storage-gen1). Hence the adl:// method, which was designed to connect to ADLS Gen1 is obsolete.
0 commit comments