[ray] Better UX for finding local modules

[SumNeuron]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
• Ray installed from (source or binary): pip
• Ray version: latest
• Python version: 3.5
• Exact command to reproduce: see linked repo

1. clone this repo
2. cd to project folder
3. jupyter-notebook <cloned-project>/jupyter/
4. just run the cells

Describe the problem

In the linked repo I provide a M.W.E. of how I envision sacred and tune would work together. Inside there is a local python module named project which contains four submodules model, utils, sacred and tune.

The model submodule has code for a simple twin embedding model in keras (simply to provide some "realism" to the MWE. This code functions and can be tested in the directory jupyter.

The sacred submodule, attempts to decouple sacred specific code as much as possible (e.g. ingredients, observers, etc). If you import the ex from this module it still runs on its own. (again, see the jupyter directory)

The tune submodule is where I envision how tune would wrap the sacred ex for hyperparameter optimization.

Most likely, as I get more familiar with tune (or for users who are) the amount of code here would expand. Nonetheless there are two simple functions, one which wraps sacred, and another which registers the wrapper and expects a hyper-parameter configuration to be passed.

When calling this function ray / tune fails as:

(pid=40079) ModuleNotFoundError: No module named 'project'

which seems to be because of this python module structure.

Notably, if all this code is put at the top level of a module, this works.

I would like to keep this structure though, and I do not see why tune should or would be opinionated about that.

Source code / logs

repo at https://gitlab.com/SumNeuron/extune

[richardliaw]

Hi @SumNeuron, can you try printing sys.path, both in the run_fn and in tune_run_ex?

[robertnishihara]

See also https://stackoverflow.com/questions/55358554/ray-tune-and-sacred-tune-cant-find-local-module.

[SumNeuron]

@richardliaw

# <repo>/project/tune/experiment.py
import ray, os, sys
from ray import tune

from ..sacred.experiment import  ex

# wrapper function over sacred experiment
def tune_run_ex(config, reporter):
    print('TUNE_RUN_EX\n\n', sys.path, '\n\n')
    ex.run(config_updates=config)
    result = ex.current_run.result
    reporter(result=result, done=True)


ray.init(ignore_reinit_error=True)
def run_fn(config):
    print('RUN_FN\n\n', sys.path, '\n\n')
    tune.register_trainable('tune_run_ex', tune_run_ex)
    tune.run_experiments({
        'demo': {
            'run': 'tune_run_ex',
            'stop': {'result': 0.1},
            'config': config,
            'resources_per_trial': {
                'cpu': 1,
                'gpu': 1
            },
            'num_samples': 1,
        }
    })

In the above code block you can see I added two print statements.

# <repo>/jupyter/Test.ipynb
import os, sys
sys.path.insert(0, '../')

import ray
from ray import tune
from project.tune.experiment import run_fn


run_fn(
    config={
        'latent_factors':  tune.grid_search([1, 2, 3])
    }
)

which results in:

# NOTE: this just runs forever

RUN_FN

 ['/home/sumner/anaconda3/lib/python3.6/site-packages/ray/pyarrow_files', '/home/sumner/Projects/extune/jupyter', '/home/sumner/anaconda3/lib/python3.6/site-packages', '/home/sumner/Projects/extune/jupyter', '/home/sumner/anaconda3/lib/python3.6/site-packages', '../', '', '/home/sumner/anaconda3/lib/python36.zip', '/home/sumner/anaconda3/lib/python3.6', '/home/sumner/anaconda3/lib/python3.6/lib-dynload', '/home/sumner/anaconda3/lib/python3.6/site-packages', '/home/sumner/anaconda3/lib/python3.6/site-packages/IPython/extensions', '/home/sumner/.ipython', '/home/sumner/anaconda3/lib/python3.6/site-packages/ray/modin'] 


== Status ==
Using FIFO scheduling algorithm.
Resources requested: 0/40 CPUs, 0/1 GPUs
Memory usage on this node: 8.4/100.0 GB

== Status ==
Using FIFO scheduling algorithm.
Resources requested: 1/40 CPUs, 1/1 GPUs
Memory usage on this node: 8.4/100.0 GB
Result logdir: /home/sumner/ray_results/demo
Number of trials: 3 ({'RUNNING': 1, 'PENDING': 2})
PENDING trials:
 - tune_run_ex_1_latent_factors=2:	PENDING
 - tune_run_ex_2_latent_factors=3:	PENDING
RUNNING trials:
 - tune_run_ex_0_latent_factors=1:	RUNNING

2019-03-27 09:33:43,349	ERROR worker.py:1780 -- Failed to unpickle actor class 'WrappedFunc' for actor ID fb1ec0b30866962cad2b5db15179befe8466c019. Traceback:
Traceback (most recent call last):
  File "/home/sumner/anaconda3/lib/python3.6/site-packages/ray/function_manager.py", line 720, in _load_actor_class_from_gcs
    actor_class = pickle.loads(pickled_class)
ModuleNotFoundError: No module named 'project'

2019-03-27 09:33:43,362	ERROR trial_runner.py:460 -- Error processing event.
Traceback (most recent call last):
  File "/home/sumner/anaconda3/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 409, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/home/sumner/anaconda3/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 314, in fetch_result
    result = ray.get(trial_future[0])
  File "/home/sumner/anaconda3/lib/python3.6/site-packages/ray/worker.py", line 2316, in get
    raise value
ray.exceptions.RayTaskError: ray_worker (pid=9892, host=deltabonn-3)
Exception: The actor with name WrappedFunc failed to be imported, and so cannot execute this method.

2019-03-27 09:33:43,381	INFO ray_trial_executor.py:178 -- Destroying actor for trial tune_run_ex_0_latent_factors=1. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.

(pid=9892) 2019-03-27 09:33:43,324	ERROR function_manager.py:723 -- Failed to load actor class %s.
(pid=9892) Traceback (most recent call last):
(pid=9892)   File "/home/sumner/anaconda3/lib/python3.6/site-packages/ray/function_manager.py", line 720, in _load_actor_class_from_gcs
(pid=9892)     actor_class = pickle.loads(pickled_class)
(pid=9892) ModuleNotFoundError: No module named 'project'

[richardliaw]

Sorry for the slow reply. Can you try moving the import into the function, like:

import ray, os, sys
from ray import tune


# wrapper function over sacred experiment
def tune_run_ex(config, reporter):
    print('TUNE_RUN_EX\n\n', sys.path, '\n\n')
	from ..sacred.experiment import  ex
    ex.run(config_updates=config)
    result = ex.current_run.result
    reporter(result=result, done=True)


ray.init(ignore_reinit_error=True)
def run_fn(config):
    print('RUN_FN\n\n', sys.path, '\n\n')
    tune.register_trainable('tune_run_ex', tune_run_ex)
    tune.run_experiments({
        'demo': {
            'run': 'tune_run_ex',
            'stop': {'result': 0.1},
            'config': config,
            'resources_per_trial': {
                'cpu': 1,
                'gpu': 1
            },
            'num_samples': 1,
        }
    })

[SumNeuron]

@richardliaw @robertnishihara 's suggestion to use a setup.py file and pip install -e is a sufficient work around...

However I personally dislike both approaches. In general python style guides encourage all import statements at the top of the file.

Having to setup and manage the module shouldn't be necessary either.

Please clone the repo and try the jupyter notebook as well as the "script.py" file.
I ask this not just because it may help in figuring out exactly why these workarounds are necessary, but because I added a fair amount of documentation throughout the functions. So it might serve as as good starting point should ray / tune want to add an example or official docs on sacred integration

I think the true solution requires the ray module to become relative / local module aware.