Running docker image headless on cluster without display

[mbillast]

❓ Questions and Help

Is it possible to run the fairembodied/habitat:latest docker image headless on a cluster without a display?
I tried it without changing any settings and got these errors:
root@f966248d58e1:/# source activate habitat
(habitat) root@f966248d58e1:/# cd habitat-api/
(habitat) root@f966248d58e1:/habitat-api# python examples/benchmark.py
2020-02-19 10:37:44,695 initializing sim Sim-v0
WARNING: Logging before InitGoogleLogging() is written to STDERR F0219 10:37:45.219249 90 WindowlessContext.cpp:111] Check failed: isNvidiaGpuReadable(eglDevId) [EGL] EGL device 0, CUDA device 0 is not readable * Check failure stack trace: *
Aborted (core dumped)

Thanks in advance for the help

[erikwijmans]

I’ve had no issue running the docker image on a headless cluster. That check failing indicates the GPU device isn’t readable, which is an odd/unexpected state for it to be in. Can you check with whoever manages the cluster?

[mbillast]

I checked with the support of the cluster and I ran nvidia-smi, which gave the following output:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... On | 00000000:AF:00.0 Off | N/A |
| 0% 30C P8 9W / 250W | 1MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

So the GPU should be readable I think, but it still gives the same error.

[mathfac]

@mbillast, what was the issue? Was Nvidia Docker installed? Thank you!

[mbillast]

@mathfac there was an issue on the cluster but it is resolved and it works now. And nvidia docker was indeed installed. Thanks for asking

[ZhuFengdaaa]

@mbillast I have exact the same problem as you. And I run my program on cluster as well. Could you share your solution please? Thank you!!

[erikwijmans]

If possible, please use a newer version of habitat. We have made EGL device selection more robust. That check above was misfiring for some versions of Docker that don't renumber the device files when mounting GPUs.

[ruddradev]

Hi @erikwijmans,

When I run habitat sim v1.5 example.py I get the below errors -

I0520 17:29:14.381590  8627 WindowlessContext.cpp:100] [EGL] Detected 2 EGL devices
F0520 17:29:14.381757  8627 WindowlessContext.cpp:117] Check failed: isNvidiaGpuReadable(eglDevId) [EGL] EGL device 0, CUDA device 0 is not readable

The strange thing is that the instance I am running it on just has one GPU, so it should not be showing 2 EGL devices. Is there a way to find out which of these two devices are the correct one and set the code to run on that? Also for the suggestion above, the code I need to run on top of Habitat 1.5 is written for for that version. Hence making modifications in Habitat-sim would be easier than changing the version.

But if that is complex, could you suggest the next version that contains the Cuda device fixes you mentioned above? (1.6 or 1.7). Thanks a lot for your help.

[erikwijmans]

You can try commenting out https://github.com/facebookresearch/habitat-sim/blob/020041d75eaf3c70378a9ed0774b5c67b9d3ce99/src/esp/gfx/WindowlessContext.cpp#L117-L119

[ruddradev]

Thank you @erikwijmans . The above remedy fixed the problem. The example files are working now.