sparse and other duck array issues

[dcherian]

Issues I've run into working with @friedrichknuth at the Pangeo meeting trying to use sparse

1. .plot() doesn't work. We need to convert to dense before plotting
2. .values doesn't work. Raises an error about explicitly calling todense()
3. align doesn't seem to work.

[shoyer]

I think it is intentional for automatic coercion to NumPy arrays to fail. Making plot() and .values work should probably require explicitly writing datarray.to_dense() first.

[dcherian]

The better way to phrase this is:

1. Should we formalize the convention that .values always return a numpy array i.e. it will call todense() implicitly and that .data will always return the underlying container: sparse/dask etc.?
2. Should we call todense() automatically in plot() the same way we do compute()?

[shoyer]

• Should we formalize the convention that .values always return a numpy array i.e. it will call todense() implicitly and that .data will always return the underlying container: sparse/dask etc.?

I think .values should either return a NumPy array or raise an exception, based upon whether or not the underlying duck array supports coercion with np.array().

We should have a separate API (maybe .to_dense() or .to_numpy_data()?) for explicitly converting into NumPy arrays. This should not be called automatically inside xarray.

Basically, we should leave the decision about whether automatic coercion is safe up to the authors of duck array libraries.

[dcherian]

👍 I think to_numpy() would be good.

[shoyer]

The main downside of to_numpy() is that it isn't obvious whether the result should be a DataArray containing a NumPy array or a raw NumPy array. In particular, pandas uses to_numpy() for converting into a raw NumPy, which is the opposite of our intent here.

[max-sixty]

I do think the pandas approach is pretty good here - don't break anything relying on .values - instead have .array to get the backing array (in whatever form it might be), and to_numpy() coercing to a raw numpy array

ref https://pandas.pydata.org/pandas-docs/stable/whatsnew/v0.24.0.html#accessing-the-values-in-a-series-or-index

[shoyer]

Xarray's .data is basically the equivalent of pandas's .array.

…

On Fri, Aug 23, 2019 at 2:44 PM Maximilian Roos ***@***.***> wrote: I do think the pandas approach is pretty good here - don't break anything relying on .values - instead have .array to get the backing array (in whatever form it might be), and to_numpy() coercing to a raw numpy array ref https://pandas.pydata.org/pandas-docs/stable/whatsnew/v0.24.0.html#accessing-the-values-in-a-series-or-index — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3245?email_source=notifications&email_token=AAJJFVUUGARF55CXIKAQHUDQGBD4NA5CNFSM4IOZW5JKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5BJCVI#issuecomment-524456277>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJJFVWSWKSJFWNUUGRO3ODQGBD4NANCNFSM4IOZW5JA> .

[khaeru]

As far as I can tell, the proposal here will require either

s = pd.Series(...)
xr.DataArray.from_series(s).to_series()

or:

xr.DataArray.from_series(s, sparse=True).to_dense().to_series()

For any code that can't guarantee sparse/non-sparse input, the first will fail sometimes, so it will always be necessary to write the latter everywhere, which IMO is unnecessarily verbose.

[fujiisoup]

Do we arrive at the consensus here for API to change the sparse or numpy array?
xref #3542

To make it sparse array, to_sparse() would be better? How about or as_sparse()?

• to_sparse() is probably consistent to todense() method
• as_sparse() sounds similar to sparse's function, e.g. sparse.as_coo

To change the backend back from sparse array, to_dense() would be better?
FYI, sparse uses todense().

I personally like as_sparse or as_numpy (or as_dense?), which sounds similar to as_type, which gives xarray object not dtype itself.

[dcherian]

I weakly prefer following the upstream API: as_sparse and todense.

though to_sparse,to_dense and to_numpy_data would be more consistent?

Also isn't the function astype not as_type?