Pyright reportPrivateImportUsage when `from datasets import load_dataset`

[lkhphuc]

Describe the bug

Pyright complains about module not exported.

Steps to reproduce the bug

Use an editor/IDE with Pyright Language server with default configuration:

from datasets import load_dataset

Expected results

No complain from Pyright

Actual results

Pyright complain below:

`load_dataset` is not exported from module "datasets"
 Import from "datasets.load" instead [reportPrivateImportUsage] 

Importing from datasets.load does indeed solves the problem but I believe importing directly from top level datasets is the intended usage per the documentation.

Environment info

• datasets version: 1.18.3
• Platform: macOS-12.2.1-arm64-arm-64bit
• Python version: 3.9.10
• PyArrow version: 7.0.0

[mariosasko]

Hi!

This issue stems from datasets having py.typed defined (see microsoft/pyright#3764 (comment)) - to avoid it, we would either have to remove py.typed (added to be compliant with PEP-561) or export the names with __all__/from .submodule import name as name.

Transformers is fine as it no longer has py.typed (removed in huggingface/transformers#18485)

WDYT @lhoestq @albertvillanova @polinaeterna

@sgugger's point makes sense - we should either be "properly typed" (have py.typed + mypy tests) or drop py.typed as Transformers did (I like this option better).

(cc @Wauplin since huggingface_hub has the same issue.)

[lhoestq]

I'm fine with dropping it, but autotrain people won't be happy @SBrandeis

[Wauplin]

(cc @Wauplin since huggingface_hub has the same issue.)

Hmm maybe we have the same issue but I haven't been able to reproduce something similar to "load_dataset" is not exported from module "datasets" message (using VSCode+Pylance -that is powered by Pyright). huggingface_hub contains a py.typed file but the package itself is actually typed. We are running mypy in our CI tests since ~3 months and so far it seems to be ok. But happy to change if it causes some issues with linters.

Also the top-level __init__.py is quite different in hfh than datasets (at first glance). We have a section at the bottom to import all high level methods/classes in a if TYPE_CHECKING block.

[mariosasko]

@Wauplin I only get the error if I use Pyright's CLI tool or the Pyright extension (not sure why, but Pylance also doesn't report this issue on my machine)

Also the top-level __init__.py is quite different in hfh than datasets (at first glance). We have a section at the bottom to import all high level methods/classes in a if TYPE_CHECKING block.

I tried to fix the issue with TYPE_CHECKING, but it still fails if py.typed is present.

[Wauplin]

@mariosasko thank for the tip. I have been able to reproduce the issue as well. I would be up for including a (huge) static __all__ variable in the __init__.py (since the file is already generated automatically in hfh) but honestly I don't think it's worth the hassle.

I'll delete the py.typed file in huggingface_hub to be consistent between HF libraries. I opened a PR here: huggingface/huggingface_hub#1329

[marcospgp]

I am getting this error in google colab today:

The code runs just fine too.