benchmark: add calibrate-n script

[RafaelGSS]

benchmark: add calibrate-n script

This script should help identify the best N
when creating/updating benchmarks

---

It's not new that our benchmark suite takes a huge amount to conclude. I have experienced that myself while writing the State of Node.js Performance 2023 and 2024.

With the help of #59174, I have measured how long it takes to conclude (of course, on my personal machine). This resulted in

Full source in: https://docs.google.com/spreadsheets/d/17iZuGR57r7ap4hFG8Tdz5Q-A7oV0cwM4MQW4q5bHA0U/edit?usp=sharing.

The problem is that some benchmarks contain a lot of configurations and an arbitrary N(the number of iterations we run between bench.start() and bench.end(n)). That said, I wrote a calibrate-n.js scripts that measure an "ideal" N for each benchmark file, taking into consideration the following constraints:

• CV (Coefficient of Variation) less than 5% - default to 30 runs to each configuration
• Overall CV (between all configurations) is less than 10%

This should help benchmark files to find the lowest N with the lowest variance and thus reduce all benchmark suite time. I'll be opening some PRs during the week, changing the N of some benchmarks.

[nodejs-github-bot]

Review requested:

• @nodejs/performance
• @nodejs/tsc

[H4ad]

Awesome tool

[H4ad]

I was thinking about this, it will be a good sanity check to get an old PR with perf improvements with the benchmark percentages and then checkout and run this tool to change the parameters to see what we will have in terms of percentages of improvements with the new parameters.

[RafaelGSS]

I was thinking about this, it will be a good sanity check to get an old PR with perf improvements with the benchmark percentages and then checkout and run this tool to change the parameters to see what we will have in terms of percentages of improvements with the new parameters.

That only works if:

• We checked out to the same state as the "perf improvement PR"
• Use the same Node.js version
• Use the same environment where the initial benchmark was created

So, it's a bit impractical. This tool was created as an attempt to find the good N as most benchmarks is using a N too big. So, in theory, it won't change much.