threadpool - lockups

[indutny]

I have serious concerns about our threadpool thingy. It seems to be quite easy to make it hang, just get some fds of blocking devices or FIFOs and issue uv_fs_write or uv_fs_open until all threads will be blocked.

I wonder if we could somehow watch their state and interrupt them with signals or something like that if they are not responding and causing stalls, after such interrupts they could be queued and moved to be run in one thread, making other worker threads non-blocking (at least).

[indutny]

cc @saghul @bnoordhuis

[indutny]

see nodejs/node-v0.x-archive#7256 (comment) for more info.

[bnoordhuis]

Sending signals doesn't strike me as a safe thing to do when the thread is executing uv_queue_work() user code.

You would need to split the thread pool in two: one for internal operations and one for executing user code, in particular because user code can change the signal mask (though you could mitigate that by saving and restoring the signal mask every time.)

The solution I explored a year or two ago was to create a large number of "threadlets" (threads with small stacks) and create more on demand.

But you will sooner or later run into the per-user process/thread limit that way and it's hard to guarantee reasonable latency, you're at the mercy of your operating system's task scheduler.

[saghul]

We probably need that bifid threadpool @bnoordhuis explored, though you'd hit limits at some point. Now I think there are 2 types of things that should run in the threadpool:

• known and short-ish blocking operations
• unknown user code

fs operations should fall into the first bucket IMHO. So, is there anything we can do to prevent uv_fs_open from being abused? The idea of stat()-ing the fd and reporting an error if it's not something we want to work with crossed my mind. Too expensive perhaps?

Since the Node report is about FIFOs, what would be the ideal way to work with those with the tools we have?

A sidenote about the threadpool: I was waiting for 0.12 to be released to propose this but I wanted to use the current unix threadpool also on Windows. IIRC it has been mentioned in the past. Not use about the performance part, but we'd gain 2 things: predictable size and a working uv_cancel, which is a good start, I guess :-)

[indutny]

Ok, could we at least consider sending signal on uv_loop_destroy()?

[saghul]

On 03/07/2014 10:06 AM, Fedor Indutny wrote:

Ok, could we at least consider sending signal on uv_loop_destroy()?

How would that solve the issue? Can you sketch out your plan? :-)

Saúl Ibarra Corretgé
bettercallsaghul.com

[indutny]

Well, I just want to interrupt threads :P If we are basically done, and don't have any ref'ed requests, or if the user is calling exit(0) - the threads should terminate ASAP.

[saghul]

On 03/07/2014 10:21 AM, Fedor Indutny wrote:

Well, I just want to interrupt threads :P If we are basically done, and
don't have any ref'ed requests, or if the user is calling |exit(0)| -
the threads should terminate ASAP.

Is node supposed to exit gracefully if exit(0) is used? (honest
question, I don't know)

So, you plan to add signal handlers on all threads in the pool? Now,
what if that signal (is SIGALRM what you have in mind?) is sent by user
code? I guess we'd also need a flag to signal that the loop is shutting
down. And I guess we need to handle non-final interrupts, ie, you get
the signal, which interrupts a file read, but the process is not going
to exit, someone just sent it.

You want this for 0.12? Sounds a bit like a band-aid...

Saúl Ibarra Corretgé
bettercallsaghul.com

[indutny]

Yeah, I'm still not sure about it... I wonder if there could be any other ways of handling it.

[saghul]

On 03/07/2014 10:51 AM, Fedor Indutny wrote:

Yeah, I'm still not sure about it... I wonder if there could be any
other ways of handling it.

That would probably work for work like blocking syscalls, but what if I
used queue_work to calculate a Fibonacci sequence or mine some Bitcoin?
;-) Not sure how we could interrupt that...

Saúl Ibarra Corretgé
bettercallsaghul.com

[indutny]

Oh, indeed. The question is, should we wait for them if there are no references?

[saghul]

Well, the fact that work is being done means there is at least one active UV_WORK request. (or a UV_FS one in case uv_fs_* was used). I don't hink we can ignore those.

Now, can node queue up arbitrary work using uv_queue_work?

In any case, here is what Python does, FWIW: sys.exit(num) will do a gracefull exit, that is, raise SystemExit, cleanup the interpreter, yadayadayada, but os._exit(num) does an abrupt exit without cleanup. If you want your process to exit no matter what, os._exit can be used.

[saghul]

Edited the above comment, note the underscore in os._exit.

[indutny]

Yeah, perhaps we can't do anything about it.

[saghul]

Can we tentatively close this Fedor? I don't see what we can do ATM.