Behavior of multi-phase initialized modules on re-import after sys.modules reset (pybind11 v3)

[ConorMacBride]

Hi,

I'm trying to understand the correct behavior and expectations for pybind11 v3.0.0+ modules that use multi-phase initialization (PEP 489). After upgrading from v2.x to v3.0.0, if I import a pybind11 module (e.g., mycode), then manually delete it from sys.modules and import it again, I get the following error:

ImportError: generic_type: type "num" is already registered!

This didn't happen under pybind11 v2.x (single-phase init), but now occurs consistently in v3.0.0. Here's a minimal reproducer:

import sys
import mycode  # OK
del sys.modules["mycode"]
import mycode  # Fails in pybind11 v3.0.0+

#include <pybind11/pybind11.h>

namespace py = pybind11;

enum mycode_e {
    ZERO = 0,
    ONE = 1
};

PYBIND11_MODULE(mycode, m)
{
    py::enum_<mycode_e>(m, "num")
        .value("zero", ZERO)
        .value("one", ONE);
}

For context, this issue surfaced while using the pytest's pytester plugin (which snapshots and restores sys.modules between test runs) to test Matplotlib's pytest plugin (matplotlib/pytest-mpl#248). Future releases of Matplotlib will be built with pybind11>=3, and we are running into issues with import matplotlib across multiple tests in pytest-mpl.

• Is this behavior expected and intentional with multi-phase initialization?
• Is this something pybind11 should/could handle more gracefully?
• Should modules defensively guard against duplicate type registrations (e.g., checking if types already exist)?
• Would adding an explicit __pybind11_version__ or __pybind11_multi_phase__ attribute help downstream tools like pytest detect modules that shouldn't be deleted from sys.modules? Or maybe there is already a way? Longer term, maybe pytester should use subinterpreters instead of resetting sys.modules. It does have a subprocess mode, but I'd rather avoid it due to its speed.

The behavior seems directly related to the multi-phase init work in #5574, so tagging @b-pass for possible input – hope that's okay!

Thanks for the great work on pybind11!

[b-pass]

I didn't know that del sys.modules[...] did this.... I am surprised that single phase init actually reloads the shared library when you do this.

At any rate, I wouldn't say that this behavior was expected (by me) or intentional.

I don't know of a way to trigger the exact same behavior (completely unloading the module) while using multi-phase init. It doesn't seem possible. We could work around it with a few hacky options (like storing the module reference somewhere and returning the same one if called again).

The easiest solution is, as you suggest, putting an attribute or something and telling anyone trying to do this to check that attribute instead. Since I suspect doing something like this is kind of rare...

[b-pass]

Potential fix in #5782.

[ConorMacBride]

Thanks @b-pass for looking into this! Your caching PR looks great! I think it would be good to preserve this old behavior provided it doesn't affect the stability of the more common import flows.

[rwgk]

#5782 was merged.