[Serve] Bad import in deployment __init__ causes deployment to restart forever #17823
Description
What is the problem?
Ray version and other system information (Python version, TensorFlow version, OS):
Master, python 3.7, Mac OS
Reproduction (REQUIRED)
from ray import serve
serve.start()
@serve.deployment
class Counter:
def __init__(self):
import does_not_exist
self.count = 0
def __call__(self, *args):
self.count += 1
return {"count": self.count}
# Deploy our class.
Counter.deploy()Prints this forever:
2021-08-13 09:34:53,515 INFO services.py:1266 -- View the Ray dashboard at http://127.0.0.1:8265
(pid=5370) 2021-08-13 09:34:55,345 INFO http_state.py:75 -- Starting HTTP proxy with name 'SERVE_PROXY_ACTOR:SERVE_CONTROLLER_ACTOR:FekFIn-node:10.103.15.138-0' on node 'node:10.103.15.138-0' listening on '127.0.0.1:8000'
2021-08-13 09:34:55,635 INFO api.py:688 -- Started Serve instance in namespace 'serve'.
2021-08-13 09:34:55,642 INFO api.py:402 -- Updating deployment 'Counter'.
(pid=5370) 2021-08-13 09:34:55,668 INFO backend_state.py:869 -- Adding 1 replicas to backend 'Counter'.
(pid=5366) INFO: Started server process [5366]
(pid=5370) 2021-08-13 09:34:55,991 ERROR controller.py:123 -- Exception updating backend state: Failed to look up actor with name 'Counter#tNDsam:SERVE_CONTROLLER_ACTOR:FekFIn'. This could because 1. You are trying to look up a named actor you didn't create. 2. The named actor died. 3. The actor hasn't been created because named actor creation is asynchronous. 4. You did not use a namespace matching the namespace of the actor.
(pid=5368) 2021-08-13 09:34:55,965 ERROR worker.py:428 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::Counter#tNDsam:SERVE_CONTROLLER_ACTOR:FekFIn:RayServeWrappedReplica.__init__ (pid=5368, ip=10.103.15.138)
(pid=5368) File "/Users/archit/anaconda3/envs/ray-py36/lib/python3.6/concurrent/futures/_base.py", line 425, in result
(pid=5368) return self.__get_result()
(pid=5368) File "/Users/archit/anaconda3/envs/ray-py36/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
(pid=5368) raise self._exception
(pid=5368) File "/Users/archit/ray/python/ray/serve/backend_worker.py", line 74, in __init__
(pid=5368) await sync_to_async(_callable.__init__)(*init_args)
(pid=5368) File "<ipython-input-1-b960fee8d0da>", line 9, in __init__
(pid=5368) ModuleNotFoundError: No module named 'does_not_exist'
2021-08-13 09:34:56,094 INFO api.py:410 -- Deployment 'Counter' is ready at `http://127.0.0.1:8000/Counter`.
(pid=5370) 2021-08-13 09:34:56,195 WARNING backend_state.py:961 -- Replica Counter#tNDsam of backend Counter failed health check, stopping it.
(pid=5370) 2021-08-13 09:34:56,303 INFO backend_state.py:869 -- Adding 1 replicas to backend 'Counter'.
(pid=5371) 2021-08-13 09:34:56,562 ERROR worker.py:428 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::Counter#IvQqVH:SERVE_CONTROLLER_ACTOR:FekFIn:RayServeWrappedReplica.__init__ (pid=5371, ip=10.103.15.138)
(pid=5371) File "/Users/archit/anaconda3/envs/ray-py36/lib/python3.6/concurrent/futures/_base.py", line 425, in result
(pid=5371) return self.__get_result()
(pid=5371) File "/Users/archit/anaconda3/envs/ray-py36/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
(pid=5371) raise self._exception
(pid=5371) File "/Users/archit/ray/python/ray/serve/backend_worker.py", line 74, in __init__
(pid=5371) await sync_to_async(_callable.__init__)(*init_args)
(pid=5371) File "<ipython-input-1-b960fee8d0da>", line 9, in __init__
(pid=5371) ModuleNotFoundError: No module named 'does_not_exist'
(pid=5370) 2021-08-13 09:34:56,629 ERROR controller.py:123 -- Exception updating backend state: Failed to look up actor with name 'Counter#IvQqVH:SERVE_CONTROLLER_ACTOR:FekFIn'. This could because 1. You are trying to look up a named actor you didn't create. 2. The named actor died. 3. The actor hasn't been created because named actor creation is asynchronous. 4. You did not use a namespace matching the namespace of the actor.
(pid=5370) 2021-08-13 09:34:56,844 WARNING backend_state.py:961 -- Replica Counter#IvQqVH of backend Counter failed health check, stopping it.
(pid=5370) 2021-08-13 09:34:56,959 INFO backend_state.py:869 -- Adding 1 replicas to backend 'Counter'.
(pid=5373) 2021-08-13 09:34:57,224 ERROR worker.py:428 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::Counter#mPHTrA:SERVE_CONTROLLER_ACTOR:FekFIn:RayServeWrappedReplica.__init__ (pid=5373, ip=10.103.15.138)
(pid=5373) File "/Users/archit/anaconda3/envs/ray-py36/lib/python3.6/concurrent/futures/_base.py", line 425, in result
(pid=5373) return self.__get_result()
(pid=5373) File "/Users/archit/anaconda3/envs/ray-py36/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
(pid=5373) raise self._exception
(pid=5373) File "/Users/archit/ray/python/ray/serve/backend_worker.py", line 74, in __init__
(pid=5373) await sync_to_async(_callable.__init__)(*init_args)
(pid=5373) File "<ipython-input-1-b960fee8d0da>", line 9, in __init__
(pid=5373) ModuleNotFoundError: No module named 'does_not_exist'
If the code snippet cannot be run by itself, the issue will be closed with "needs-repro-script".
- I have verified my script runs in a clean environment and reproduces the issue.
- I have verified the issue also occurs with the latest wheels.