forkserver slow to start because of __main__ preloading issue

[emaxx-google]

Bug report

Bug description:

When switching a Python project from the default multiprocessing fork mode to forkserver, I've noticed the processes in the pool got significantly slower to start. Specifying preload modules didn't help.

Debugging internals of multiprocessing showed that most of the time (50 ms in each process in my test) is spent in _fixup_main_from_path(data['init_main_from_path']):

cpython/Lib/multiprocessing/spawn.py

Line 246 in eae9d7d

_fixup_main_from_path(data['init_main_from_path'])

This seemed surprising because __main__ was mentioned in the preload parameter, however I've noticed that forkserver.py tries to populate the main_path parameter from spawn.get_preparation_data():

cpython/Lib/multiprocessing/forkserver.py

Lines 149 to 151 in eae9d7d

	desired_keys = {'main_path', 'sys_path'}
	data = spawn.get_preparation_data('ignore')
	main_kws = {x: y for x, y in data.items() if x in desired_keys}

However, the latter only writes the path to the init_main_from_path parameter:

cpython/Lib/multiprocessing/spawn.py

Line 202 in eae9d7d

d['init_main_from_path'] = os.path.normpath(main_path)

The end effect is that the __main__ module wasn't preloaded in practice, and every child process had to re-run the main script. Unless I'm missing something, the logic in forkserver.py needs to get main_path from the value of init_main_from_path?

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-137996: Fix path to __main__ in forkserver preload logic #138048

[XChaitanyaX]

I am new here, is this a good first issue for me to work on?
@emaxx-google can you please share the testcase or the benchmark script.

[picnixz]

I am new here, is this a good first issue for me to work on?

I don't think so. If you want a good first issue, check for the ones with the easy label.

[emaxx-google]

I have a PR already, but as I'm not deeply familiar with the internals I'm not 100% sure about it.

[duaneg]

See also issue #126631 and comments, and the associated pull request #135295.

[emaxx-google]

See also issue #126631 and comments, and the associated pull request #135295.

Thanks for pointing this out! The implementation change looks almost identical, and you have a regression test already; do you know what's blocking that change from landing?

[duaneg]

Thanks for pointing this out! The implementation change looks almost identical, and you have a regression test already; do you know what's blocking that change from landing?

No worries at all, and yeah, our independent fixes are basically the same, which is nice 🙂

I don't think anything is blocking the change as such, but AIUI the maintainers and core devs in general are overworked and have to prioritise where they spend their time and attention. Before you raised this issue it was a fairly academic fix: it had been broken for many years, seemingly without anyone noticing. It is useful to know it has caused a noticeable performance regression in real-world use!