rules_python and linking with libpython

[canada-geek]

Hello,

Our C++ project produces a collection of dynamic libraries. Along with each library we use swig to generate a python interface for the libraries.

Swig produces 2 files for each library: a python file, and a C++ source file with the wrapper code.

As you can imagine, this swig C++ file contains many calls to the python C API, and as such the resulting swig dynamic library needs to be linked with some version of libpython3.X.so.

Our project needs to be able to build in multiple environments (including customer site), including x86_64 vs. aarch64 (both target and host). Because of this, we are leveraging rules_python to decouple the build definitions from the python toolchain and the actual python version/path being used.

In order to link with the python library, I (perhaps naively) added @rules_python//python/cc:current_py_cc_libs to the deps of the cc_library(). However, the resulting command line did not have -lpython3.X.

Upon further examination of the @rules_python//python/cc:current_py_cc_libs target, I found that:

• it has a CcInfo provider
• the CcInfo compilation context has the headers, however the include paths for it are listed as -iquote, which defeats swig's use of #include <Python.h> (not a quoted include) in its generated code. Also adding @rules_python//python/cc:current_py_cc_headers solves this, however it would be nice if libs was sufficient in this case
• the CcInfo linking context listed 2 libraries, but both are designated as alwayslink=False.
• it seems counterintuitive (and contrary to toolchain decoupling) to be forced to add the library manually to linkopts

What would be the idiomatic way to address this? One thought I had was to have two targets in rules_python: the existing one, and another with alwayslink=True to force linking; then it would be up to me to choose which is best for my environment.

I'm using bazel 8.3.1 and rules_python 1.1.0. Our workspace uses bzlmod.

Would it help if I move ahead to a newer version of rules_python?

Thanks for your time and attention.

[rickeylev]

libs target causes -iquote, while headers target doesn't

Hm, weird. There's not intentional, so must be some idiosyncrasy of the cc rules. No idea why one target would use iquote and the other isystem. (isystem should be used, since <Python.h> is the idiomatic way to include the python headers).

cc info listed two libraries, both designated as alwayslink=False
add alwayslink=True variant
have to manually add -lpython to linkopts

Yeah, the link settings are a quagmire. AFAIU, The basic problem is:

If it adds -lpython, then it goes into DT_NEEDED, and now you're forced to using the dynamic linker for finding it. This means, if one wants to manually dynamically load it using dlopen (or otherwise provide the symbols later some other way), you can no longer do so.

alwayslink=true causes a similar problem -- it forces the final binary (be it a cc_binary or shared library) to always link against it.

Since consumers can add extra e.g. link opts, but can't remove them, I thought it best to omit those settings.

Another reason was it wasn't clear to me what the correct -l<name> values were. I can't remember exactly, but it looked like there was plain libpython and some versioned variants (I can't recall if it was imply libpythonX.Y.so and/or libpythonX.Y.so.N names), and I think windows uses slightly different nameing ("XY" instead of "X.Y"?).

In any case, I couldn't find docs that explained things, so also thought it best to omit it so users could add the necessary values. Clarity, references, and expertise in this area is much welcomed.

Would it help if I move ahead to a newer version of rules_python?

Maybe? I don't recall any major changes sinces 1.1, but things like -iquote are pretty subtle that nobody may have noticed.

[canada-geek]

@rickeylev Thanks for your response and explanations, much appreciated.

I understand the need to keep alwayslink=False. For my use case, I indeed need to force the dynamic linking and inclusion in DT_NEEDED.

Can you (or anyone) suggest a way to do this without forcing a hard-code of the library name in linkopts? I will continue experimenting to see if I can find a way to achieve this.

I will also move to latest rules_python, we don't have a compelling reason to stay at the current version.

[rickeylev]

By hardcoding, i'm guessing you mean adding -lpython3.12 (the version specific part)?

Two thoughts come to mind:

(1) Use a select(). You'll have to enumerate the condition -> values. If that set is small, then this is an easy and teneable option. e.g. linkopts = select({":is_py_312": ["-lpython3.12"], ...}), where is_py_312 is e.g. config_setting(<match @rules_python//python/config_settings:python_version_major_minor = 3.12>)

(2) Custom rule that reads the settings and populates CcInfo link opts. Something like this:

def _libpython_linkopts_impl(ctx):
  runtime = ctx.toolchains["string-from-below"].py3_runtime
  libname = "python{major}.{minor}{ft}".format(
      major = runtime.interpreter_version_info.major,
      minor = runtime.interpreter_version_info.minor,
      ft = "t" if "t" in runtime.abi_flags else ""
  )
  return [CcInfo(linking_context=cc_common.create_linking_context(
    user_link_flags = ["-l{}".format(libname)]
  ))]

libpython_linkopts = rule(
  implementation = _libpython_linkopts_impl,
  toolchains = ["@rules_python//python:toolchain_type"]
)

That should add the appropriate -l name based on the python version and free threading-ness. To make it multi-platform, use ctx.target_platform_has_constraint, or use select to pass a value into the rule that tells the e.g. os.

(the above is pretty boiler-platey, so maybe we should add e.g. //python/cc:dynamic_linking_linkopts or some such)

HTH

[canada-geek]

Many thanks, this I think gives me what I need for now.