Archiver for musl is not `ar` when in Docker image for a different architecture

[samestep]

When I use cc natively on Linux or macOS, cc::Build::get_archiver always returns ar when I'm building for either x86 musl or ARM musl. However, when I try to do the same build in a rust Docker image, and the --platform is not my native platform, it sometimes attempts other names for the archiver program (like musl-ar or aarch64-linux-musl-ar) which cause the build to fail.

This forces me to set a TARGET_AR variable in my Dockerfile if I depend on crates like wasmtime that transitively depend on crates like zstd-sys; see this Stack Overflow question for such an example, which is itself boiled down from gradbench/gradbench#233 that was my real use case.

See this GitHub repository which includes a full code example to reproduce the issue. As also written in that README.md, here are the different values I see returned by cc::Build::get_archiver in various contexts:

• ARM macOS
  • native: "ar"
  • native targeting x86 musl: "ar"
  • native targeting ARM musl: "ar"
  • Docker targeting x86 musl: "musl-ar"
  • Docker targeting ARM musl: "ar"
• x86 Linux
  • native: "ar"
  • native targeting x86 musl: "ar"
  • native targeting ARM musl: "ar"
  • Docker targeting x86 musl: "ar"
  • Docker targeting ARM musl: "aarch64-linux-musl-ar"

[NobodyXu]

It might be that musl-ar contains some special workaround necessary for the target?

[madsmtm]

This is kind of the expected behaviour, we check if you're cross-compiling, and attempt to use an archiver more suited for that task:

cc-rs/src/lib.rs

Line 3288 in fcf940e

} else if self.get_is_cross_compile()? {

The reason you're seeing something different on your local machine might be because you have AR, RUSTC_LINKER or similar set locally?

[madsmtm]

Or actually, I think we choose musl-ar/aarch64-linux-musl-ar over ar because they exist:

cc-rs/src/lib.rs

Lines 3301 to 3302 in fcf940e

	if Command::new(&target_p).output().is_ok() {
	chosen = target_p;

[madsmtm]

Why do the musl-ar/aarch64-linux-musl-ar binaries in your Docker container not work?

[samestep]

@NobodyXu

It might be that musl-ar contains some special workaround necessary for the target?

But then I have two questions:

1. why does everything work correctly when I set ENV TARGET_AR=ar in my Dockerfile?
2. how am I meant to install musl-ar? As shown in that linked Stack Overflow question, it is not installed as part of apt-get install musl-tools.

---

@madsmtm

This is kind of the expected behaviour, we check if you're cross-compiling, and attempt to use an archiver more suited for that task:

I thought that too at first, but get_is_cross_compile just checks whether TARGET and HOST are the same, right?

cc-rs/src/lib.rs

Lines 3551 to 3558 in 15fe112

	fn get_is_cross_compile(&self) -> Result<bool, Error> {
	let target = self.get_raw_target()?;
	let host: Cow<'_, str> = match &self.host {
	Some(h) => Cow::Borrowed(h),
	None => Cow::Owned(self.getenv_unwrap_str("HOST")?),
	};
	Ok(host != target)
	}

The archiver is something other than ar in only 2/10 of the cases I listed above, whereas TARGET and HOST differ in 8/10 of the cases, as you can see in the additional output I've just added to the GitHub README I linked:

• ARM macOS
  • native:

    TARGET = Some(aarch64-apple-darwin)
    HOST = Some(aarch64-apple-darwin)
    

  • native targeting x86 musl:

    TARGET = Some(x86_64-unknown-linux-musl)
    HOST = Some(aarch64-apple-darwin)
    

  • native targeting ARM musl:

    TARGET = Some(aarch64-unknown-linux-musl)
    HOST = Some(aarch64-apple-darwin)
    

  • Docker targeting x86 musl:

    TARGET = Some(x86_64-unknown-linux-musl)
    HOST = Some(x86_64-unknown-linux-gnu)
    

  • Docker targeting ARM musl:

    TARGET = Some(aarch64-unknown-linux-musl)
    HOST = Some(aarch64-unknown-linux-gnu)
    

• x86 Linux
  • native:

    TARGET = Some(x86_64-unknown-linux-gnu)
    HOST = Some(x86_64-unknown-linux-gnu)
    

  • native targeting x86 musl:

    TARGET = Some(x86_64-unknown-linux-musl)
    HOST = Some(x86_64-unknown-linux-gnu)
    

  • native targeting ARM musl:

    TARGET = Some(aarch64-unknown-linux-musl)
    HOST = Some(x86_64-unknown-linux-gnu)
    

  • Docker targeting x86 musl:

    TARGET = Some(x86_64-unknown-linux-musl)
    HOST = Some(x86_64-unknown-linux-gnu)
    

  • Docker targeting ARM musl:

    TARGET = Some(aarch64-unknown-linux-musl)
    HOST = Some(aarch64-unknown-linux-gnu)
    

The reason you're seeing something different on your local machine might be because you have AR, RUSTC_LINKER or similar set locally?

I don't. But to be clear, I'm more confused about the difference between the linux/amd64 and linux/arm64 versions of the Docker image when I run it on machines of different architectures; what happens when I run locally is less important.

Or actually, I think we choose musl-ar/aarch64-linux-musl-ar over ar because they exist:

cc-rs/src/lib.rs

Lines 3301 to 3302 in fcf940e

	if Command::new(&target_p).output().is_ok() {
	chosen = target_p;

No; they don't exist in those Docker images, which is why the build fails when I don't add ENV TARGET_AR=ar.

Why do the musl-ar/aarch64-linux-musl-ar binaries in your Docker container not work?

As mentioned above and in the Stack Overflow answer, they don't exist in the Docker container.

[samestep]

@madsmtm just to clarify, are you able to reproduce the behavior in Docker using the instructions I gave in that GitHub repository?

To run in Docker targeting x86 musl:

docker build --platform linux/amd64 . -t foo-x86 && docker run --rm foo-x86

To run in Docker targeting ARM musl:

docker build --platform linux/arm64 . -t foo-arm && docker run --rm foo-arm

[NobodyXu]

I'm more confused about the difference between the linux/amd64 and linux/arm64 versions of the Docker image when I run it on machines of different architectures;

No; they don't exist in those Docker images, which is why the build fails when I don't add ENV TARGET_AR=ar.

cc-rs/src/lib.rs

Lines 3301 to 3302 in fcf940e

	if Command::new(&target_p).output().is_ok() {
	chosen = target_p;

This check is for cross compilation, it seems that we are not doing cross compilation here but rather just changing HOST by using different docker image

Is that correct?

[NobodyXu]

I suspect that maybe the docker image comes with environment vsriables set for AR, because I can't find any musl- in our code, however checking the dockerfile didn't find any

Maybe you can run env inside the docker containers to see if any eng var is set?

[madsmtm]

No; they don't exist in those Docker images, which is why the build fails when I don't add ENV TARGET_AR=ar.

Huh. But that's the entire reason for that check?

are you able to reproduce the behavior in Docker

Didn't yet, I don't have Docker installed locally, and it's kind of a hazzle so I was hoping to do without.

I can't find any musl- in our code, however checking the dockerfile didn't find any

See prefix_for_target.

[madsmtm]

If you could provide the output of env in the container like @NobodyXu said, as well as the output of which musl-ar && which ar && which aarch64-linux-musl-ar, that'd be helpful; otherwise I'll try to get Docker set up and test it locally.

[samestep]

@madsmtm

If you could provide the output of env in the container like @NobodyXu said, as well as the output of which musl-ar && which ar && which aarch64-linux-musl-ar, that'd be helpful; otherwise I'll try to get Docker set up and test it locally.

Sure thing! Similar to before, I ran this command first:

$ docker build --platform linux/amd64,linux/arm64 . -t foo

Here are the outputs when running on my M1 MacBook:

$ docker run --platform linux/amd64 --rm foo env
PATH=/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=0207e7628bc5
RUSTUP_HOME=/usr/local/rustup
CARGO_HOME=/usr/local/cargo
RUST_VERSION=1.84.1
HOME=/root
$ docker run --platform linux/amd64 --rm foo which ar
/usr/bin/ar
$ docker run --platform linux/amd64 --rm foo which musl-ar
$ docker run --platform linux/amd64 --rm foo which aarch64-linux-musl-ar
$ docker run --platform linux/arm64 --rm foo env
PATH=/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=562d3a2a58a4
RUSTUP_HOME=/usr/local/rustup
CARGO_HOME=/usr/local/cargo
RUST_VERSION=1.84.1
HOME=/root
$ docker run --platform linux/arm64 --rm foo which ar
/usr/bin/ar
$ docker run --platform linux/arm64 --rm foo which musl-ar
$ docker run --platform linux/arm64 --rm foo which aarch64-linux-musl-ar

And here they are when running on my x86 Linux machine:

$ docker run --platform linux/amd64 --rm foo env
PATH=/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=bf34022b9b8e
RUSTUP_HOME=/usr/local/rustup
CARGO_HOME=/usr/local/cargo
RUST_VERSION=1.84.1
HOME=/root
$ docker run --platform linux/amd64 --rm foo which ar
/usr/bin/ar
$ docker run --platform linux/amd64 --rm foo which musl-ar
$ docker run --platform linux/amd64 --rm foo which aarch64-linux-musl-ar
$ docker run --platform linux/arm64 --rm foo env
PATH=/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=0f32dd3d45c9
RUSTUP_HOME=/usr/local/rustup
CARGO_HOME=/usr/local/cargo
RUST_VERSION=1.84.1
HOME=/root
$ docker run --platform linux/arm64 --rm foo which ar
/usr/bin/ar
$ docker run --platform linux/arm64 --rm foo which musl-ar
$ docker run --platform linux/arm64 --rm foo which aarch64-linux-musl-ar

As you can see, all the which commands print no output and return exit code 1, except for which ar.

[samestep]

@NobodyXu

This check is for cross compilation, it seems that we are not doing cross compilation here but rather just changing HOST by using different docker image

Is that correct?

Correct, I think! Because passing --platform to docker run uses emulation to change the host architecture. But, it's still cross-compilation anyways, right? Since, e.g., x86_64-unknown-linux-musl is still different from x86_64-unknown-linux-gnu, and aarch64-unknown-linux-musl is still different from aarch64-unknown-linux-gnu. Unless I misunderstand?

[madsmtm]

Thanks for the output, that is indeed weird. I'm out of time for today, but I'll take a look during the weekend (if @NobodyXu doesn't get to it first ;)).

But, it's still cross-compilation anyways, right? Since, e.g., x86_64-unknown-linux-musl is still different from x86_64-unknown-linux-gnu, and aarch64-unknown-linux-musl is still different from aarch64-unknown-linux-gnu.

Yeah, we're still cross-compiling here.

[NobodyXu]

Correct, I think! Because passing --platform to docker run uses emulation to change the host architecture. But, it's still cross-compilation anyways, right? Since, e.g., x86_64-unknown-linux-musl is still different from x86_64-unknown-linux-gnu, and aarch64-unknown-linux-musl is still different from aarch64-unknown-linux-gnu. Unless I misunderstand?

While it's technically a cross compilation, cc-rs cannot detect it as rustc host has been changed with the target.