Detect nodes going down and reassigning work in @parallel

[ViralBShah]

The @parallel for could use some simple fault tolerance, where iterations just get scheduled on other processors if a processor disappears.

This could be simple and easy to do, before we do any fancy database backed stuff.

Assigning to Stephan since he said he wanted to do some compiler stuff. This could possibly qualify..

[StefanKarpinski]

I think this is way bigger than just an issue to be "fixed" or "implemented". This is something we all need to figure out the right approach to doing. It is kind of the key to elastic parallelism.

[JeffBezanson]

This is not in the compiler. And it will only work depending on what the parallel for loop does. If it uses distributed arrays, making it fault tolerant is not so easy.

I think the thing to do here is make sure the system has some basic robustness: if a node goes down, everything else should survive and keep working, but you might get errors if you try to do something that depends on the dead node. This has the immediate consequence that if nothing depends on the dead node, you've simply succeeded at removing a processor.

[StefanKarpinski]

This is why, in my mind, "elastic" and "fault tolerant" go together: removing a node intentionally and having a node fail and then removing it from the computation reactively are very similar. For real fault tolerance, the trick would be to ensure that even if the node responsible for a piece of a computation fails, the computation can be completed by another node eventually. Really want to hear Parry's take on this...

[ViralBShah]

I feel that just allowing nodes to go down without bringing down the rest of the system is useful. For a particular class of problems such as Monte Carlo stuff, which are likely to be the first parallel problems, it will also be sufficient.

-viral

On 15-Oct-2011, at 1:52 PM, Stefan Karpinski wrote:

This is why, in my mind, "elastic" and "fault tolerant" go together: removing a node intentionally and having a node fail and then removing it from the computation reactively are very similar. For real fault tolerance, the trick would be to ensure that even if the node responsible for a piece of a computation fails, the computation can be completed by another node eventually. Really want to hear Parry's take on this...

Reply to this email directly or view it on GitHub:
#217 (comment)

[StefanKarpinski]

That's definitely a start. For anything more complex than that, however, we're really going to need some form of fault recovery.

[JeffBezanson]

The key first step here is to decide what to do with processor ids. The set of processors will have to be a vector instead of just an integer, and nprocs is the length of that vector. We need to change everywhere that uses something like 1:np. Then we need to decide whether to reuse processor ids.

[StefanKarpinski]

Does hostname + pid on that host cut it? Or should we just assign a UUID upon joining a computation? Then the set of processors will have to be just that — a set. Can it be a GlobalObject, this set of processors?

[6e441f9c]

Would the very basic tolerance be an easier first step or not? Simply reverting to single-threaded mode after a node failure would be an improvement. I want to use local parallel processingon a 4-core server to make my interactive session quicker to respond; I am OK with getting a message and restarting the threads, but having the session hang is suboptimal.

I guess some easily checkpointable computation could also benefit right now from running a monitor on ID=1 processor that restarts computation from last known data on crash...