Auto merge of #76680 - Julian-Wollersberger:nongeneric_ensure_sufficient_stack, r=jyn514

bors · bors · commit b01326ab033e · 2020-09-21T17:32:57.000Z
Make `ensure_sufficient_stack()` non-generic, using cargo-llvm-lines Inspired by [this blog post](https://blog.mozilla.org/nnethercote/2020/08/05/how-to-speed-up-the-rust-compiler-some-more-in-2020/) from `@nnethercote,` I used [cargo-llvm-lines](https://github.com/dtolnay/cargo-llvm-lines/) on the rust compiler itself, to improve it's compile time. This PR contains only one low-hanging fruit, but I also want to share some measurements. The function `ensure_sufficient_stack()` was monomorphized 1500 times, and with it the `stacker` and `psm` crates, for a total of 1.5% of all llvm IR lines. With some trickery I convert the generic closure into a dynamic one, and thus all that code is only monomorphized once. # Measurements Getting these numbers took some fiddling with CLI flags and I [modified](https://github.com/Julian-Wollersberger/cargo-llvm-lines/blob/master/src/main.rs#L115) cargo-llvm-lines to read from a folder instead of invoking cargo. Commands I used: ``` ./x.py clean RUSTFLAGS="--emit=llvm-ir -C link-args=-fuse-ld=lld -Z self-profile=profile" CARGOFLAGS_BOOTSTRAP="-Ztimings" RUSTC_BOOTSTRAP=1 ./x.py build -i --stage 1 library/std # Then manually copy all .ll files into a folder I hardcoded in cargo-llvm-lines in main.rs#L115 cd ../cargo-llvm-lines cargo run llvm-lines ``` The result is this list (see [first 500 lines](https://github.com/Julian-Wollersberger/cargo-llvm-lines/blob/master/llvm-lines-rustc-before.txt) ), before the change: ``` Lines Copies Function name ----- ------ ------------- 16894211 (100%) 58417 (100%) (TOTAL) 2223855 (13.2%) 502 (0.9%) rustc_query_system::query::plumbing::get_query_impl::{{closure}} 1331918 (7.9%) 1287 (2.2%) hashbrown::raw::RawTable<T>::reserve_rehash 774434 (4.6%) 12043 (20.6%) core::ptr::drop_in_place 294170 (1.7%) 499 (0.9%) rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl 245410 (1.5%) 1552 (2.7%) psm::on_stack::with_on_stack 210311 (1.2%) 1 (0.0%) rustc_target::spec::load_specific 200962 (1.2%) 513 (0.9%) rustc_query_system::query::plumbing::get_query_impl 190704 (1.1%) 1 (0.0%) rustc_middle::ty::query::<impl rustc_middle::ty::context::TyCtxt>::alloc_self_profile_query_strings 180272 (1.1%) 468 (0.8%) rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory 177396 (1.1%) 114 (0.2%) rustc_query_system::query::plumbing::force_query_impl 161134 (1.0%) 445 (0.8%) rustc_query_system::dep_graph::graph::DepGraph<K>::with_anon_task 141551 (0.8%) 186 (0.3%) rustc_query_system::query::plumbing::incremental_verify_ich 110191 (0.7%) 7 (0.0%) rustc_middle::ty::context::_DERIVE_rustc_serialize_Decodable_D_FOR_TypeckResults::<impl rustc_serialize::serialize::Decodable<__D> for rustc_middle::ty::context::TypeckResults>::decode::{{closure}} 108590 (0.6%) 420 (0.7%) core::ops::function::FnOnce::call_once 88488 (0.5%) 21 (0.0%) rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_previous_green 86368 (0.5%) 1 (0.0%) rustc_middle::ty::query::stats::query_stats 85654 (0.5%) 3973 (6.8%) <&T as core::fmt::Debug>::fmt 84475 (0.5%) 1 (0.0%) rustc_middle::ty::query::Queries::try_collect_active_jobs 81220 (0.5%) 862 (1.5%) <hashbrown::raw::RawIterHash<T> as core::iter::traits::iterator::Iterator>::next 77636 (0.5%) 54 (0.1%) core::slice::sort::recurse 66484 (0.4%) 461 (0.8%) <hashbrown::raw::RawIter<T> as core::iter::traits::iterator::Iterator>::next ``` All `.ll` files together had 4.4GB. After my change they had 4.2GB. So a few percent less code LLVM has to process. Hurray! Sadly, I couldn't measure an actual wall-time improvement. Watching YouTube while compiling added to much noise... Here is the top of the list after the change: ``` 16460866 (100%) 58341 (100%) (TOTAL) 1903085 (11.6%) 504 (0.9%) rustc_query_system::query::plumbing::get_query_impl::{{closure}} 1331918 (8.1%) 1287 (2.2%) hashbrown::raw::RawTable<T>::reserve_rehash 777796 (4.7%) 12031 (20.6%) core::ptr::drop_in_place 551462 (3.4%) 1519 (2.6%) rustc_data_structures::stack::ensure_sufficient_stack::{{closure}} ``` Note that the total was reduced by 430 000 lines and `psm::on_stack::with_on_stack` has disappeared. Instead `rustc_data_structures::stack::ensure_sufficient_stack::{{closure}}` appeared. I'm confused about that one, but it seems to consist of inlined calls to `rustc_query_system::*` stuff. Further note the other two big culprits in this list: `rustc_query_system` and `hashbrown`. These two are monomorphized many times, the query system summing to more than 20% of all lines, not even counting code that's probably inlined elsewhere. Assuming compile times scale linearly with llvm-lines, that means a possible 20% compile time reduction. Reducing eg. `get_query_impl` would probably need a major refactoring of the qery system though. _Everything_ in there is generic over multiple types, has associated types and passes generic Self arguments by value. Which means you can't simply make things `dyn`. --------------------------------------- This PR is a small step to make rustc compile faster and thus make contributing to rustc less painful. Nonetheless I love Rust and I find the work around rustc fascinating :)
diff --git a/Cargo.lock b/Cargo.lock
@@ -4468,9 +4468,9 @@ checksum = "a8f112729512f8e442d81f95a8a7ddf2b7c6b8a1a6f509a95864142b30cab2d3"
 
 [[package]]
 name = "stacker"
-version = "0.1.11"
+version = "0.1.12"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a92bc346006ae78c539d6ab2cf1a1532bc657b8339c464877a990ec82073c66f"
+checksum = "21ccb4c06ec57bc82d0f610f1a2963d7648700e43a6f513e564b9c89f7991786"
 dependencies = [
  "cc",
  "cfg-if",
diff --git a/compiler/rustc_data_structures/Cargo.toml b/compiler/rustc_data_structures/Cargo.toml
@@ -26,7 +26,7 @@ rustc_index = { path = "../rustc_index", package = "rustc_index" }
 bitflags = "1.2.1"
 measureme = "0.7.1"
 libc = "0.2"
-stacker = "0.1.11"
+stacker = "0.1.12"
 tempfile = "3.0.5"
 
 [dependencies.parking_lot]
