rustup fails with symlink RUSTUP_HOME/toolchains

[jo-so]

Problem

% rustup update
error: I/O Error: Too many levels of symbolic links (os error 40)

BTW: This message isn't very helpful.

I'm having a symlink at ~/.rustup/toolchains pointing to ~/.cache/rustup/toolchains

Steps

1. mkdir ~/rustup-test && cd ~/.rustup && cp -r downloads tmp update-hashes settings.toml ~/rustup-test
2. ln -s ~/.rustup/toolchains ~/rustup-test/
3. RUSTUP_HOME=$HOME/rustup-test rustup update

Possible Solution(s)

No response

Notes

This happens since the update from

   nightly-x86_64-unknown-linux-gnu updated - rustc 1.78.0-nightly (3b1717c05 2024-03-10) (from rustc 1.78.0-nightly (2d24fe591 2024-03-09))
   stable-x86_64-unknown-linux-gnu unchanged - rustc 1.76.0 (07dca489a 2024-02-04)

to

    nightly-x86_64-unknown-linux-gnu updated - rustc 1.78.0-nightly (3cbb93223 2024-03-13) (from rustc 1.78.0-nig
htly (3b1717c05 2024-03-10))
   stable-x86_64-unknown-linux-gnu unchanged - rustc 1.76.0 (07dca489a 2024-02-04)

% RUSTUP_HOME=$HOME/rustup-test strace -Zk -e t=openat rustup update
10:27:49.009219 openat(AT_FDCWD</home/joerg/git/rustup>, "/home/joerg/kein_Backup/rustup-test/toolchains", O_RDONLY|O_NOFOLLOW|O_CLOEXEC) = -1 ELOOP (Too many levels of symbolic links) <0.000013>
 > /usr/lib/x86_64-linux-gnu/libc.so.6(__open64+0x51) [0xf7861]
 > /home/joerg/.cache/cargo/bin/rustup(std::sys::unix::fs::File::open_c+0xe8) [0x511148]
 > /home/joerg/.cache/cargo/bin/rustup(std::fs::OpenOptions::_open+0x12a) [0x5391fa]
 > /home/joerg/.cache/cargo/bin/rustup(rustup::toolchain::toolchain::Toolchain::exists+0x172) [0x3fb192]
 > /home/joerg/.cache/cargo/bin/rustup(rustup::toolchain::toolchain::Toolchain::new+0x34) [0x3fac74]
 > /home/joerg/.cache/cargo/bin/rustup(rustup::toolchain::distributable::DistributableToolchain::new+0x3f) [0x3d92af]
 > /home/joerg/.cache/cargo/bin/rustup(<core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::next+0xf80) [0x4e8550]
 > /home/joerg/.cache/cargo/bin/rustup(rustup::config::Cfg::list_channels+0x9a) [0x4e692a]
 > /home/joerg/.cache/cargo/bin/rustup(rustup::cli::rustup_mode::update+0x3b6) [0x4a38c6]
 > /home/joerg/.cache/cargo/bin/rustup(rustup::cli::rustup_mode::main+0x1b2d) [0x47e09d]
 > /home/joerg/.cache/cargo/bin/rustup(rustup_init::main+0x1262) [0xeea32]
 > /home/joerg/.cache/cargo/bin/rustup(std::sys_common::backtrace::__rust_begin_short_backtrace+0x3) [0xec8c3]
 > /home/joerg/.cache/cargo/bin/rustup(main+0x414) [0xef3c4]
 > /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_init_first+0x8a) [0x276ca]
 > /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x27785]
 > /home/joerg/.cache/cargo/bin/rustup(_start+0x29) [0xec7f7]
error: I/O Error: Too many levels of symbolic links (os error 40)

Rustup version

rustup 1.27.0 (bbb9276d2 2024-03-08)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.79.0-nightly (2f090c30d 2024-03-23)`

Installed toolchains

Default host: x86_64-unknown-linux-gnu
rustup home:  /home/joerg/.rustup

installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu (default)

active toolchain
----------------

nightly-x86_64-unknown-linux-gnu (default)
rustc 1.79.0-nightly (2f090c30d 2024-03-23)

[jo-so]

BTW: This bug leaves you with a state where you can't do anything with rustup, no update, no compile of any crate (e.g. rustup itself) to debug.

[rami3l]

@jo-so Thanks a lot for filing this issue!

I guess this is a duplicate of #3344, so I suggest we continue the discussion over there. Please feel free to correct me and reopen this issue if that's not the case.

[jo-so]

@rami3l I don't think so. #3344 discusses about symlinks for downloads and tmp. This bug report is about toolchains. And it worked since last week or so, while #3344 is open for a longer time.

[rami3l]

@jo-so Okay, I'll keep both issues open but those two issues do seem related. Unfortunately since this is caused by self-referential symlinks (created presumably in another rustup run other than the current one), your strace isn't very helpful.

I'll find some time to investigate into both issues.

[jo-so]

Here's a git-bisect run. A good version was 1.25.0, and the culprit is 2c9297d.

# bad: [1c612cf65439901200f89c0123dbdf9dcae5e539] fix(ci): fix file paths in CI-generated `*.sha256` files on *nix
# good: [976d72a1920a0266936320afc63b5e651888359f] (changelog): Fix typo
git bisect start 'origin/master' '1.25.0'
# skip: [44c7d94079d19dacdf63fd90cf16297df6293843] Merge pull request #3338 from hi-rustin/rustin-patch-clippy-tip
git bisect skip 44c7d94079d19dacdf63fd90cf16297df6293843
# good: [060b8f9e3f5a8a810b49108aa57cee2e784ee104] Merge pull request #3054 from MoSal/master
git bisect good 060b8f9e3f5a8a810b49108aa57cee2e784ee104
# bad: [d63b6e5f7608369934a4774611af9a83104d3a78] Address `#[warn(clippy::useless_conversion)]`
git bisect bad d63b6e5f7608369934a4774611af9a83104d3a78
# good: [de9a9f08c8b33f58541aa3df0ee80c5244bd5354] Merge pull request #3252 from hi-rustin/rustin-patch-actions-run-test
git bisect good de9a9f08c8b33f58541aa3df0ee80c5244bd5354
# good: [41b631c778d4806f2739e81fc7f4d7b765b8c8b9] Merge pull request #3333 from workingjubilee/use-mutex-const-new
git bisect good 41b631c778d4806f2739e81fc7f4d7b765b8c8b9
# bad: [8a704172114133095086c11b649afed6a64b3d95] CI support for loongarch64-unknown-linux-gnu
git bisect bad 8a704172114133095086c11b649afed6a64b3d95
# good: [395e49270394680dd03c74f69dcfedc6c3283ab5] Suggest right toolchain when running clippy
git bisect good 395e49270394680dd03c74f69dcfedc6c3283ab5
# bad: [2c9297d51a077d9a364e98ad8ebaee2301cd0c0d] Rework Toolchain model and drop relative file path overrides
git bisect bad 2c9297d51a077d9a364e98ad8ebaee2301cd0c0d
# good: [46812bf38a66c51558d4036c630d6640ead45492] Merge pull request #3335 from xfix/toml-0.7.3
git bisect good 46812bf38a66c51558d4036c630d6640ead45492
# good: [8662c33cc983a8fc0690ebb536cca91e027af094] Merge pull request #3343 from cuishuang/master
git bisect good 8662c33cc983a8fc0690ebb536cca91e027af094
# good: [4b29b79af7fe09cc7911528257241be5fbba25c6] Merge pull request #3287 from rbtcollins/tracing
git bisect good 4b29b79af7fe09cc7911528257241be5fbba25c6
# first bad commit: [2c9297d51a077d9a364e98ad8ebaee2301cd0c0d] Rework Toolchain model and drop relative file path overrides

[jo-so]

@jo-so Okay, I'll keep both issues open but those two issues do seem related. Unfortunately since this is caused by self-referential symlinks (created presumably in another rustup run other than the current one), your strace isn't very helpful.

I'll find some time to investigate into both issues.

It's not about self-referential symlinks. As you can see in the strace the flag O_NOFOLLOW gets passed to the openat, which causes not even one symlink is crossed.

With a self compiled version you get

% RUSTUP_FORCE_ARG0=rustup RUSTUP_HOME=/home/joerg/kein_Backup/rustup-test strace -Zk -e t=openat target/debug/rustup-init update
11:59:57.099680 openat(AT_FDCWD</home/joerg/git/rustup>, "/home/joerg/kein_Backup/rustup-test/toolchains", O_RDONLY|O_NOFOLLOW|O_CLOEXEC) = -1 ELOOP (Too many levels of symbolic links) <0.000032>
 > /usr/lib/x86_64-linux-gnu/libc.so.6(__open64+0x51) [0xf7861]
 > /home/joerg/git/rustup/target/debug/rustup-init(std::sys::pal::unix::fs::File::open_c+0xfc) [0x10d489c]
 > /home/joerg/git/rustup/target/debug/rustup-init(std::fs::OpenOptions::_open+0x8b) [0x10c73fb]
 > /home/joerg/git/rustup/target/debug/rustup-init(std::fs::OpenOptions::open+0x73) [0x4f4223]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup::utils::raw::open_dir+0x64) [0x33d3c4]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup::toolchain::toolchain::Toolchain::exists+0x296) [0x1dbe16]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup::toolchain::toolchain::Toolchain::new+0xa8) [0x1db848]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup::toolchain::distributable::DistributableToolchain::new+0x95) [0x25bfe5]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup::config::Cfg::list_channels::{{closure}}+0x80) [0x2f8ba0]
 > /home/joerg/git/rustup/target/debug/rustup-init(core::iter::adapters::map::map_try_fold::{{closure}}+0x69) [0x3637e9]
 > /home/joerg/git/rustup/target/debug/rustup-init(core::iter::adapters::filter::filter_try_fold::{{closure}}+0x121) [0x1a2f21]
 > /home/joerg/git/rustup/target/debug/rustup-init(core::iter::adapters::filter_map::filter_map_try_fold::{{closure}}+0x14c) [0x2e7bac]
 > /home/joerg/git/rustup/target/debug/rustup-init(core::iter::traits::iterator::Iterator::try_fold+0x10f) [0x34e11f]
 > /home/joerg/git/rustup/target/debug/rustup-init(<core::iter::adapters::filter_map::FilterMap<I,F> as core::iter::traits::iterator::Iterator>::try_fold+0x4f) [0x2e692f]
 > /home/joerg/git/rustup/target/debug/rustup-init(<core::iter::adapters::filter::Filter<I,P> as core::iter::traits::iterator::Iterator>::try_fold+0x47) [0x1a1ac7]
 > /home/joerg/git/rustup/target/debug/rustup-init(<core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold+0x40) [0x361230]
 > /home/joerg/git/rustup/target/debug/rustup-init(<core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold+0x4f) [0x30611f]
 > /home/joerg/git/rustup/target/debug/rustup-init(<core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::next+0x24) [0x305e84]
 > /home/joerg/git/rustup/target/debug/rustup-init(<alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter+0x53) [0x1f22b3]
 > /home/joerg/git/rustup/target/debug/rustup-init(alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter+0x1e) [0x1fcc1e]
 > /home/joerg/git/rustup/target/debug/rustup-init(<alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter+0x29) [0x2000d9]
 > /home/joerg/git/rustup/target/debug/rustup-init(<core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}+0x22) [0x1b8302]
 > /home/joerg/git/rustup/target/debug/rustup-init(core::iter::adapters::try_process+0x74) [0x306a44]
 > /home/joerg/git/rustup/target/debug/rustup-init(<core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter+0x27) [0x1b8287]
 > /home/joerg/git/rustup/target/debug/rustup-init(core::iter::traits::iterator::Iterator::collect+0xe) [0x36283e]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup::config::Cfg::list_channels+0x106) [0x19f216]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup::config::Cfg::update_all_channels+0x3f) [0x19f2df]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup::cli::common::update_all_channels+0x54) [0x18d5e4]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup::cli::rustup_mode::update+0x43f) [0x2343ef]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup::cli::rustup_mode::main+0xd62) [0x226412]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup_init::run_rustup_inner+0x38b) [0x1813bb]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup_init::run_rustup+0x1a2) [0x180ef2]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup_init::maybe_trace_rustup+0xd) [0x180d3d]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup_init::main::{{closure}}+0xe) [0x180a4e]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup::currentprocess::with::{{closure}}+0x26c) [0x1822ec]
 > /home/joerg/git/rustup/target/debug/rustup-init(std::thread::local::LocalKey<T>::try_with+0xdc) [0x181cdc]
 > /home/joerg/git/rustup/target/debug/rustup-init(std::thread::local::LocalKey<T>::with+0x31) [0x181bd1]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup::currentprocess::with+0x85) [0x182065]
 > /home/joerg/git/rustup/target/debug/rustup-init(rustup_init::main+0x42) [0x180d22]
 > /home/joerg/git/rustup/target/debug/rustup-init(core::ops::function::FnOnce::call_once+0xb) [0x18250b]
 > /home/joerg/git/rustup/target/debug/rustup-init(std::sys_common::backtrace::__rust_begin_short_backtrace+0xe) [0x183bae]
 > /home/joerg/git/rustup/target/debug/rustup-init(std::rt::lang_start::{{closure}}+0x11) [0x183c21]
 > /home/joerg/git/rustup/target/debug/rustup-init(std::rt::lang_start_internal+0x441) [0x10c2301]
 > /home/joerg/git/rustup/target/debug/rustup-init(std::rt::lang_start+0x3a) [0x183bfa]
 > /home/joerg/git/rustup/target/debug/rustup-init(main+0x1e) [0x1816ae]
 > /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_init_first+0x8a) [0x276ca]
 > /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x27785]
 > /home/joerg/git/rustup/target/debug/rustup-init(_start+0x21) [0x1806d1]
error: I/O Error: Too many levels of symbolic links (os error 40)
11:59:57.159481 +++ exited with 1 +++

Which points to

rustup/src/utils/raw.rs

Lines 45 to 54 in 0c501d5

	#[cfg(not(windows))]
	pub fn open_dir(p: &Path) -> std::io::Result<File> {
	use std::fs::OpenOptions;
	use std::os::unix::fs::OpenOptionsExt;
	let mut options = OpenOptions::new();
	options.read(true);
	options.custom_flags(libc::O_NOFOLLOW);
	options.open(p)
	}

that adds the NOFOLLOW flag

contrary to Toolchain::exists()

rustup/src/toolchain/toolchain.rs

Lines 70 to 74 in 0c501d5

	let opened = OpenOptions::default()
	.read(true)
	.follow(true)
	.open_dir_at(&parent_dir, base_name);
	Ok(opened.is_ok())

[rami3l]

@jo-so Wow, many thanks for your help with the investigation! It's true that with libc::O_NOFOLLOW, ELOOP will have a slight different meaning. Thanks again for the reminder!

I have no idea why this option flag is enabled though, and this patch was included before I was even there. Maybe @rbtcollins can provide more context on this one?

---

Considering the Windows side of the same function, it doesn't seem to have the same semantics:

rustup/src/utils/raw.rs

Lines 40 to 43 in 0c501d5

	let mut options = OpenOptions::new();
	options.read(true);
	options.custom_flags(FILE_FLAG_BACKUP_SEMANTICS);
	options.open(p)

FILE_FLAG_BACKUP_SEMANTICS
0x02000000
The file is being opened or created for a backup or restore operation. The system ensures that the calling process overrides file security checks when the process has SE_BACKUP_NAME and SE_RESTORE_NAME privileges [..]
You must set this flag to obtain a handle to a directory. A directory handle can be passed to some functions instead of a file handle.

OTOH the only caller of this function is Toolchain::exists, and I don't see a particular reason why we can't follow a symlink for this purpose.

[rami3l]

@Joso I just did some more research on this one.

The snippet that you mentioned above actually came from remove_dir_all, and the same code copied over to Rustup is indeed the root cause of #3344 😓

    fn open_dir(p: &Path) -> io::Result<fs::File> {
        let mut options = OpenOptions::new();
        options.read(true);
        options.custom_flags(libc::O_NOFOLLOW);
        options.open(p)
    }

https://github.com/XAMPPRocky/remove_dir_all/blob/c3927b9d5c0194a0b5115bcff4a3fffbc8b97f1f/src/_impl/unix.rs#L26-L31

For the reference, remove_dir_all seems to consistently drop an error when the dir is a symlink (which might not be the expected behavior on our side anyway). However on Windows it generates a custom error, whereas on Unix it returns ELOOP:

    fn open_dir(p: &Path) -> Result<File> {
        let mut options = OpenOptions::new();
        options.read(true);
        options.custom_flags(FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT);
        let maybe_dir = options.open(p)?;
        if maybe_dir.metadata()?.is_symlink() {
            return Err(io::Error::new(
                io::ErrorKind::Other,
                "Path is a directory link, not directory",
            ));
        }
        Ok(maybe_dir)
    }

https://github.com/XAMPPRocky/remove_dir_all/blob/c3927b9d5c0194a0b5115bcff4a3fffbc8b97f1f/src/_impl/win.rs#L55-L67