"pip install ray" requires gcc due to setproctitle

[robertnishihara]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16
• Ray installed from (source or binary): pip
• Ray version: nightly wheels
• Python version: 3.6

I actually was installing Ray through the autoscaler with

setup_commands:
    - sudo apt-get update
    # - sudo apt-get install -y cmake pkg-config build-essential autoconf curl libtool unzip flex bison python
    - wget https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh || true
    - bash Anaconda3-5.0.1-Linux-x86_64.sh -b -p $HOME/anaconda3 || true
    - echo 'export PATH="$HOME/anaconda3/bin:$PATH"' >> ~/.bashrc

    - pip install boto3==1.4.8 cython==0.27.3
    - pip install https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.5.3-cp36-cp36m-manylinux1_x86_64.whl

It failed with

Building wheels for collected packages: setproctitle
  Running setup.py bdist_wheel for setproctitle: started
  Running setup.py bdist_wheel for setproctitle: finished with status 'error'
  Complete output from command /home/ubuntu/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-5s9_wmul/setproctitle/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpljn3n0ynpip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  running build_ext
  building 'setproctitle' extension
  creating build
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/src
  gcc -pthread -B /home/ubuntu/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DSPT_VERSION=1.1.10 -DHAVE_SYS_PRCTL_H=1 -I/home/ubuntu/anaconda3/include/python3.6m -c src/setproctitle.c -o build/temp.linux-x86_64-3.6/src/setproctitle.o
  unable to execute 'gcc': No such file or directory
  error: command 'gcc' failed with exit status 1
  
  ----------------------------------------
  Running setup.py clean for setproctitle
  Failed building wheel for setproctitle
Failed to build setproctitle
Installing collected packages: redis, flatbuffers, setproctitle, funcsigs, ray
  Running setup.py install for setproctitle: started
    Running setup.py install for setproctitle: finished with status 'error'
    Complete output from command /home/ubuntu/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-5s9_wmul/setproctitle/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-7r48zj4i-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'setproctitle' extension
    creating build
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/src
    gcc -pthread -B /home/ubuntu/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DSPT_VERSION=1.1.10 -DHAVE_SYS_PRCTL_H=1 -I/home/ubuntu/anaconda3/include/python3.6m -c src/setproctitle.c -o build/temp.linux-x86_64-3.6/src/setproctitle.o
    unable to execute 'gcc': No such file or directory
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "/home/ubuntu/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-5s9_wmul/setproctitle/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-7r48zj4i-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-5s9_wmul/setproctitle/
You are using pip version 9.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
NodeUpdater: Error updating (Exit Status 1) 

Basically, pip installing ray requires gcc to build setproctitle, which is a bit unfortunate. This seems like it could cause a lot of installation failures.

Is there good a way to optionally pull certain dependencies but not "require" them?

cc @ericl

[atumanov]

hmm, actually , fwiw, when I encountered this error message I resolved with just a pip install setproctitle. I.e., if we just add setproctitle to the set of pip installed packages, it might be sufficient.

[ericl]

Why is that trying to run bdist_wheel on setproctitle? A normal install shouldn't involve this right.

[robertnishihara]

I think it's basically pulling the files and running the setup.py, which in this case appears to be compiling things. https://github.com/dvarrazzo/py-setproctitle/blob/master/setup.py

[ericl]

That's not how pip install usually work right? There should be a binary package. When I pip install setproctitle I don't see this behavior.

…

On Mon, Nov 26, 2018, 3:09 PM Robert Nishihara ***@***.***> wrote: I think it's basically pulling the files and running the setup.py, which in this case appears to be compiling things. https://github.com/dvarrazzo/py-setproctitle/blob/master/setup.py — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3406 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA6SnDSLABsk_nFep0LVdHKUyGPa9SLks5uzHSVgaJpZM4Yzw5w> .

[robertnishihara]

I think it depends on what the package authors choose to upload to PyPI. E.g., if you upload a wheel, then no compilation should be necessary. However, if you just upload a tar of your repository, then it will try to run setup.py. In this case, it looks to me like it is just a copy of the repository https://pypi.org/project/setproctitle/#files.

Ok, I just tried it with Python 2 since I didn't already have it installed for setproctitle and I was worried that the compiled stuff would be cached. And it did run setup.py

$ /Users/rnishihara/anaconda2/bin/pip install setproctitle
Collecting setproctitle
  Using cached https://files.pythonhosted.org/packages/5a/0d/dc0d2234aacba6cf1a729964383e3452c52096dc695581248b548786f2b3/setproctitle-1.1.10.tar.gz
Building wheels for collected packages: setproctitle
  Running setup.py bdist_wheel for setproctitle ... done
  Stored in directory: /Users/rnishihara/Library/Caches/pip/wheels/e6/b1/a6/9719530228e258eba904501fef99d5d85c80d52bd8f14438a3
Successfully built setproctitle
Installing collected packages: setproctitle
Successfully installed setproctitle-1.1.10

If I uninstall it and reinstall it, I see just

$ /Users/rnishihara/anaconda2/bin/pip install setproctitle
Collecting setproctitle
Installing collected packages: setproctitle
Successfully installed setproctitle-1.1.10

[ericl]

Well, one option is to do the same thing as we do for psutil -- log a big warning if it's not installed.

There's no way to try catch installing these types of dependencies during setup?

[robertnishihara]

That's probably a good idea. I'm not aware of any way to cleanly try installing these dependencies during setup. You could always manually call subprocess.call(['pip', 'install', 'ray']) though it feels wrong.

[snmhaines]

What is the neat solution to this? I just tried installing ray[debug] and it failed because of the same stumbling block with setproctitle.

[robertnishihara]

@snmhaines you may need to do something like sudo apt-get install -y build-essential if you're on Ubuntu (or something similar to install gcc on MacOS).

Down the road, we should ideally just compile and ship setproctitle with Ray. cc @pcmoritz

[snmhaines]

Thanks for the reply Robert – I was trying to avoid that in order to get the cluster launching quickly. What sort of debugging facilities does ray[debug] provide? I have another question for you: when Ray forms the cluster, is there a way of getting the redis and cluster public IP addresses into a variable that I can use to streamline the start-up and initialization of Ray?

[robertnishihara]

@snmhaines setproctitle isn't really necessary. It is useful for changing the names of the worker processes so you can see which process is which from top.

Regarding getting the cluster address in a variable, I assume you're asking about how to parse the output of ray start? If so, you can hardcode the port that redis uses so that you can just use the fixed value, e.g., ray start --redis-port=6379. Presumably you will also know the address of the machine that you're on, but if you want to determine it programmatically, you can do import ray; ray.services.get_node_ip_address() in python, and then combine that with the 6379.

[alexcoca]

I also found that running the equivalent of sudo apt-get install -y build-essential on Linux Manjaro in a conda environment did not help - setproctitle build was still an issue (despite having installed libgcc). The solution has been to run conda install -c anaconda gxx_linux-64 first.