Unable to set up developer environment on Mac M1

[KMFODA]

System Info

As MacBooks with the M1 chip need tensorflow-macos installed instead of tensorflow>=2.3 (as listed in the setup.py) file trying to set up a developer environment on a M1 MacBook produces the following error:

ERROR: Could not find a version that satisfies the requirement tensorflow>=2.3; extra == "dev" (from transformers[dev]) (from versions: none)
ERROR: No matching distribution found for tensorflow>=2.3; extra == "dev"

Is there any way around this? I tried replacing with tensorflow-macos but that creates a myriad of other issues when trying to set up the developer environment.

transformers 4.21.0
python 3.9.12

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

pip install -e ".[dev]"

Expected behavior

Developer environment should be installed without errors.

[ArthurZucker]

Hey! Thanks for the issue. I managed to get it work doing the following :

1. Replace tensorflow with tensorflow-macos in the setup.py file.
2. Install particular dependencies manually

brew install llvm
conda install -c apple tensorflow-deps
python -m pip install tensorflow-macos
python -m pip install tensorflow-metal
pip3 install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu
pip install fugashi==1.1.2a6
pip install numba
brew install cmake 
brew install rust 
conda install -c conda-forge onnxruntime

3. Just run pip install -e ".[dev]" and it should work :).

[KMFODA]

Thanks so much for the prompt response @ArthurZucker. This is great although when I run the final pip install -e ".[dev]" command I get the following error:

ERROR: Could not find a version that satisfies the requirement tensorflow-text; extra == "dev" (from transformers[dev]) (from versions: none)
ERROR: No matching distribution found for tensorflow-text; extra == "dev"

when I remove tensor flow-text from setup.py the pip install -e ".[dev]" runs fine. Did you experience a similar issue?

[ArthurZucker]

At the time (~2 month ago), I did not. But it seems like it is a pretty known issue mentioned here. You should apparently build tensorflow-text from source or use the python wheel made available 👍🏻 (installing with pip install tensorflow-text does not work either)

[KMFODA]

Amazing that seems to work for now. Thanks for the help!

[ArEnSc]

sadly this guide no longer works as
using conda 3.9.11
conda install -c apple tensorflow-deps
python -m pip install tensorflow-macos
python -m pip install tensorflow-metal also fails

[ArEnSc]

The issue is you want to use miniforge (community version of conda) and not the conda...

[LWprogramming]

Following up here with what I've tried so far with my issue, in case it's useful for anyone in the future:

• tensorflow-text doesn't have a Python 3.11 prebuilt wheel, so I used 3.10 for everything.
• Install miniforge as mentioned here instead of conda, because it has tensorflow-deps and conda doesn't.
• follow instructions in here, in an active conda environment.
• Remove decord manually from setup.py because it's not actively maintained anymore according to this, and my use case shouldn't need decord.
• While conda environment is active, create virtual environment and try installing:
  python3.10 -m venv venv && source venv/bin/activate && pip install --upgrade pip && pip uninstall transformers && pip install -e ".[dev]"
• I then got a resolution error:

ERROR: Cannot install transformers and transformers[dev]==4.30.0.dev0 because these package versions have conflicting dependencies.

ERROR: Cannot install transformers and transformers[dev]==4.30.0.dev0 because these package versions have conflicting dependencies.

The conflict is caused by:
    transformers[dev] 4.30.0.dev0 depends on jax!=0.3.2, <=0.3.6 and >=0.2.8; extra == "dev"
    flax 0.6.9 depends on jax>=0.4.2
    transformers[dev] 4.30.0.dev0 depends on jax!=0.3.2, <=0.3.6 and >=0.2.8; extra == "dev"
    flax 0.6.8 depends on jax>=0.4.2
# ... many similar lines of text

• Installing with pip install -e ".[quality]" instead of dev worked, which is fine for my use case because I'm not modifying anything with Jax but not a complete solution unfortunately.