Poor multi-threaded allocation performance

[m-mueller678]

This program, when run on linux reports a time of 57ms, but when run on hermit takes 13s.

fn main() {
    let t1 = std::time::Instant::now();
    let threads: Vec<_> = (0..128)
        .map(|tid| {
            std::thread::spawn(|| {
                for aid in 0..1_000_1000 {
                    Box::new([0u64; 8]);
                }
            })
        })
        .collect();
    for t in threads {
        t.join().unwrap();
    }
    dbg!(t1.elapsed());
}

I am measuring this on a processor with 16 cores, 32 threads. I am using qemu with -smp 32.

I have looked into it a bit and found that the default global allocator forwards calls to the kernel allocator, which is Talc wrapped in a mutex, so poor multithreaded performance is to be expected. Given this huge slowdown, I think this should either not be the default allocator, or there should be some visibly placed guidance on how to obtain better allocation performance. Unfortunately, I could get neither jemalloc nor mimalloc to run on hermit out of the box.

[jounathaen]

Yes, this issue is currently present. This is caused by an allocator issue, which we currently track in the kernel repo: #1968 and we have also created an upstream issue: SFBdragon/talc#44

If you are keen on experimenting, you can try another allocator, which probably performs better for the moment: #1940

I'm closing this issue, as we best track it in the aforementioned kernel issue. Feel free to add any information or reopen if necessary.

[mkroening]

#1982 will resolve the single-thread regressions (#1968).

After that is done, we should measure this again.
Two things might slow us down here, though:

1. A lot of task switching might happen here. I am not sure about our performance compared to Linux, so we may want to use the same number of threads and cores to avoid measuring that.
2. The single-threaded scheduler. Since there are no inherently parallel allocator crates that would be good candidates, as far as I know, we should do some thread-local sharding of memory. fix: rework TLS, fix alignments #1983 should help with that since it allows faster TCB access in a follow-up PR compared to querying core-local storage.