Add `LazilyLoadedLibrary` type to `Libdl`

[staticfloat]

I was mulling over the latest issue regarding JLLs eagerly loading libraries in their __init__() functions (in this case, MKL gets broken if CSL_jll loads libgomp first, since it conflicts with symbols loaded from libiomp5) and I think I might have a solution. The way I want to fix that is to have BB analyze libraries for their dependencies, and then instead of eagerly loading them at __init__() time, load them at first ccall or dlsym() time. The issue is that the code that exists in the wild right now is:

using libfoo_jll

ccall((libfoo, :fooify), ...)

If I could change the libfoo in there to libfoo(), I could run a function that JIT loads all the dependent libraries (on a library-by-library basis, no longer eargerly loading all libraries within a single JLL because I don't know which bindings the user is going to want to use) and then returns the SONAME. Unfortunately, that breaks basically every user code ever.

However, I had an idea; what if we created a LazilyLoadedLibrary type that contained within it a list of other LazilyLoadedLibrary types which were its dependencies, and the ccall() lowering (as well as dlsym() and other methods) was taught to load those? Then the work that the JLLs have to do is just to export these objects for use. I think it might be possible to maintain pretty good compatibility with the ecosystem if we change a type under their noses, rather than changing all of their syntax.

This would eliminate the vast majority of JLL __init__() methods, should speed up launch time, will fix some behavioral bugs, and moves us closer to JLLs being inert pieces of data, rather than code.

[KristofferC]

and I think I might have a solution.

I'll just mention that maybe this is not really a 100% solution. Even with libraries lazily loaded you still can't escape the fact that libgomp and libiomp5 cannot coexist in the same process (I think). So even with the laziness proposed here you will still not be able to call someting from CSL_jll and expect having MKL to work.

[staticfloat]

Yes, that's true, this just gets us back to v1.8 levels of compatibility where you have to load MKL first and refrain from using anything that requires the libgomp-specific bits. There's really no solution there as far as I can tell. :(

[vtjnash]

The problem isn't calling dlopen either, it is doing it with custom special code in every unique special JLL file. If there were simply an array of things in a magic variable and Base did the actual loading, this would be much less of a problem. Then we could even transition to making them read as a TOML file, instead of executing a Julia file and building a whole Julia module for each one, and apply our existing work towards caching TOML to make that very efficient.

[KristofferC]

The problem isn't calling dlopen either, it is doing it with custom special code in every unique special JLL file. If there were simply an array of things in a magic variable and Base did the actual loading, this would be much less of a problem.

Most jlls do kind of the same thing in their __init__ so replacing that with some magic variable that contains a list of libraries to dlopen which then that triggers Julia itself to actually open them just after it has loaded that jll seems like it would not be too hard?

[staticfloat]

If there were simply an array of things in a magic variable and Base did the actual loading, this would be much less of a problem.

I think we're stepping toward that with this proposal, we can transition the vast majority of the JLLs that exist into (essentially) a list of LazilyLoadedLibrary objects, and then Libdl can be the one responsible for loading the libraries just in time.

There remain some JLLs that will require __init__() methods to do things like set environment variables and whatnot, but those are a small fraction.

[KristofferC]

There remain some JLLs that will require init() methods to do things like set environment variables

Even that could be done by Base, no? You only really need an init if you need truly arbitrary code execution.

[vtjnash]

__init__() methods to do things like set environment variables

This is likely unsafe, in that the libc standard defines this to be unsafe, and might cause other libraries (esp. openblas which starts threads) to segfault. This is not advisable.

[staticfloat]

Even that could be done by Base, no? You only really need an init if you need truly arbitrary code execution.

With enough special syntax, yes. But the question is, what is enough special syntax? To give some concrete examples, in order of increasing complexity:

• PMIx: sets an environment variable to artifact_dir. This is probably the simplest thing, just add an environment mapping that points to the prefix of the artifact. This could be easily implemented by having a set of environment variables that get mapped to the artifact, e.g. artifact_envs = Set(["PMIX_PREFIX"]), and those get set to the artifact directory.
• Fontconfig: Sets a few environment variables to point to resolved paths within the artifact, but also supports being overridden by environment variables passed in from above. So extending the above, we could instead store a dictionary and have tokens we can replace, e.g. envs = Dict("FONTCONFIG_FILE" => "${artifact_dir}/etc/fonts/fonts.conf"). We can then have overrideable_envs = Set(["FONTCONFIG_FILE"]) too, I suppose. Note that I simplified things a bit here compared to what the __init__() is actually doing, but this is sufficient for this discussion.
• FLINT: has to make a ccall() to register runtime functions with FLINT after opening. True code execution.
• CUDA: Tim suffers no imitators, all who stare upon the visage of CUDA's __init__() method surrender to uncontrollable awe and despair.

[staticfloat]

This is likely unsafe, in that the libc standard defines this to be unsafe, and might cause other libraries (esp. openblas which starts threads) to segfault. This is not advisable.

I'm not really sure we have an alternative. Some libraries require environment variables to be set before you can call their initialization functions. For executables we can indeed come up with a better API that applies the environment variables to Cmd objects (which is what Git.jl does) but for poorly-written libraries, I think setting the environment variables in __init__() is actually the best choice, as it sets them as early as possible, before user code has a chance to start up more threads and calling other code.

[staticfloat]

To make this proposal slightly more concrete, this is what I could put in Libdl:

"""
    LazyLibrary(name, flags = <default dlopen flags>,
                dependencies = LazyLibrary[], on_load_callback = nothing)

Represents a lazily-loaded library that opens itself and its dependencies on first usage
in a `dlopen()`, `dlsym()`, or `ccall()` usage.  While this structure contains the
ability to run arbitrary code on first load via `on_load_callback`, we caution that this
should be used sparingly, as it is not expected that `ccall()` should result in large
amounts of Julia code being run.
"""
mutable struct LazyLibrary
    # Name and flags to open with
    const name::String
    const flags::UInt32

    # Dependencies that must be loaded before we can load
    const dependencies::Vector{LazyLibrary}

    # Function that gets called once upon initial load with the pointer as an argument
    const on_load_callback::Union{Nothing,Function}

    # Pointer that we eventually fill out upon first `dlopen()`
    @atomic handle::Ptr{Cvoid}
    function LazyLibrary(name, flags = _default_rtld_flags, dependencies = LazyLibrary[], on_load_callback = nothing)
        return new(
            String(name),
            UInt32(flags),
            collect(dependencies),
            on_load_callback,
            C_NULL,
        )
    end
end

function dlopen(ll::LazyLibrary)
    # Only load once
    hdl = @atomic ll.handle
    if hdl != C_NULL
        return hdl
    end

    # Ensure that all dependencies are loaded
    for dep in ll.dependencies
        dlopen(dep)
    end

    # Load our library
    hdl = dlopen(ll.name, ll.flags)
    @atomic ll.handle = hdl

    # Invoke our on load callback, if it exists
    if ll.on_load_callback !== nothing
        ll.on_load_callback(hdl)
    end
    return hdl
end
dlsym(ll::LazyLibrary, args...; kwargs...) = dlsym(dlopen(ll), args...; kwargs...)

This works just fine for explicit dlopen() usage:

flags = Libd._default_rtld_flags
libgcc_s = LazyLibrary("/home/sabae/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/bin/../lib/julia/libgcc_s.so.1", flags, LazyLibrary[], hdl -> @info("Loaded libcc_s", hdl))
libgfortran = LazyLibrary("/home/sabae/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/bin/../lib/julia/libgfortran.so.5", flags, [libgcc_s], hdl -> @info("Loaded libgfortran", hdl))
libopenblas = LazyLibrary("/home/sabae/.julia/artifacts/3f4ae12d938ec5fab4f2d701b8d0baa28d7dea16/lib/libopenblas.so", flags, [libgfortran], hdl -> @info("Loaded libopenblas", hdl))
libsdpa = LazyLibrary("/home/sabae/.julia/artifacts/605f995afb4e0f23f8c2c10977af13526b52dddb/lib/libsdpa.so", flags, [libgfortran, libopenblas], hdl -> @info("Loaded libsdpa", hdl))

julia> dlopen(libsdpa)
┌ Info: Loaded libcc_s
└   hdl = Ptr{Nothing} @0x0000555953df54e0
┌ Info: Loaded libgfortran
└   hdl = Ptr{Nothing} @0x0000555953e008a0
┌ Info: Loaded libopenblas
└   hdl = Ptr{Nothing} @0x00005559533375c0
┌ Info: Loaded libsdpa
└   hdl = Ptr{Nothing} @0x0000555953d16e10
Ptr{Nothing} @0x0000555953d16e10

julia> dlopen(libsdpa)
Ptr{Nothing} @0x0000555953d16e10

Next up, I need to integrate this into ccall() lowering so that ccall() emits code to do a dlopen(::LazyLibrary) and use the return value in the follow-on code, similar to what 5cba88c does.

[staticfloat]

With Valentin and Gabriel's help, I managed to make a draft implementation: #50074