Skip to content

benchmark: adjust configuration for string-decoder bench #59187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

RafaelGSS
Copy link
Member

@RafaelGSS RafaelGSS commented Jul 23, 2025

benchmark: adjust configuration for string-decoder bench

According to #59186 this benchmark file takes 6 hours to complete a full benchmark/compare.js script (60 runs in total) and these regression tests unrealitics to do between Node.js releases. By using calibrate-n scripts I could find a better N also ajusting some bench configs. e.g: avoid dead code elimination by V8.

Refs: #59186

Results from calibrate-n.js

Overall average CV: 3.57%
  ✓ Overall CV is below 5% and no configuration has CV above 10%
  → New best n: 400000 with average CV: 3.57%

✓ Found optimal n=400000 (Overall CV=3.57% < 5% and no configuration has CV > 10%)

Final CV for each configuration:
  {"n":400000,"chunkLen":16,"inLen":32,"encoding":"ascii"}: 5.11%
  {"n":400000,"chunkLen":256,"inLen":32,"encoding":"ascii"}: 7.24%
  {"n":400000,"chunkLen":1024,"inLen":32,"encoding":"ascii"}: 2.60%
  {"n":400000,"chunkLen":16,"inLen":128,"encoding":"ascii"}: 2.64%
  {"n":400000,"chunkLen":256,"inLen":128,"encoding":"ascii"}: 3.88%
  {"n":400000,"chunkLen":1024,"inLen":128,"encoding":"ascii"}: 3.33%
  {"n":400000,"chunkLen":16,"inLen":1024,"encoding":"ascii"}: 2.54%
  {"n":400000,"chunkLen":256,"inLen":1024,"encoding":"ascii"}: 5.85%
  {"n":400000,"chunkLen":1024,"inLen":1024,"encoding":"ascii"}: 4.53%
  {"n":400000,"chunkLen":16,"inLen":32,"encoding":"utf8"}: 3.51%
  {"n":400000,"chunkLen":256,"inLen":32,"encoding":"utf8"}: 4.27%
  {"n":400000,"chunkLen":1024,"inLen":32,"encoding":"utf8"}: 5.20%
  {"n":400000,"chunkLen":16,"inLen":128,"encoding":"utf8"}: 2.98%
  {"n":400000,"chunkLen":256,"inLen":128,"encoding":"utf8"}: 1.68%
  {"n":400000,"chunkLen":1024,"inLen":128,"encoding":"utf8"}: 2.14%
  {"n":400000,"chunkLen":16,"inLen":1024,"encoding":"utf8"}: 1.36%
  {"n":400000,"chunkLen":256,"inLen":1024,"encoding":"utf8"}: 1.15%
  {"n":400000,"chunkLen":1024,"inLen":1024,"encoding":"utf8"}: 1.12%
  {"n":400000,"chunkLen":16,"inLen":32,"encoding":"base64-utf8"}: 3.61%
  {"n":400000,"chunkLen":256,"inLen":32,"encoding":"base64-utf8"}: 3.86%
  {"n":400000,"chunkLen":1024,"inLen":32,"encoding":"base64-utf8"}: 4.45%
  {"n":400000,"chunkLen":16,"inLen":128,"encoding":"base64-utf8"}: 3.34%
  {"n":400000,"chunkLen":256,"inLen":128,"encoding":"base64-utf8"}: 4.50%
  {"n":400000,"chunkLen":1024,"inLen":128,"encoding":"base64-utf8"}: 5.32%
  {"n":400000,"chunkLen":16,"inLen":1024,"encoding":"base64-utf8"}: 1.65%
  {"n":400000,"chunkLen":256,"inLen":1024,"encoding":"base64-utf8"}: 2.55%
  {"n":400000,"chunkLen":1024,"inLen":1024,"encoding":"base64-utf8"}: 3.67%
  {"n":400000,"chunkLen":16,"inLen":32,"encoding":"base64-ascii"}: 5.55%
  {"n":400000,"chunkLen":256,"inLen":32,"encoding":"base64-ascii"}: 3.76%
  {"n":400000,"chunkLen":1024,"inLen":32,"encoding":"base64-ascii"}: 2.42%
  {"n":400000,"chunkLen":16,"inLen":128,"encoding":"base64-ascii"}: 2.14%
  {"n":400000,"chunkLen":256,"inLen":128,"encoding":"base64-ascii"}: 4.13%
  {"n":400000,"chunkLen":1024,"inLen":128,"encoding":"base64-ascii"}: 3.57%
  {"n":400000,"chunkLen":16,"inLen":1024,"encoding":"base64-ascii"}: 3.01%
  {"n":400000,"chunkLen":256,"inLen":1024,"encoding":"base64-ascii"}: 2.71%
  {"n":400000,"chunkLen":1024,"inLen":1024,"encoding":"base64-ascii"}: 3.61%
  {"n":400000,"chunkLen":16,"inLen":32,"encoding":"utf16le"}: 3.76%
  {"n":400000,"chunkLen":256,"inLen":32,"encoding":"utf16le"}: 5.35%
  {"n":400000,"chunkLen":1024,"inLen":32,"encoding":"utf16le"}: 7.25%
  {"n":400000,"chunkLen":16,"inLen":128,"encoding":"utf16le"}: 2.68%
  {"n":400000,"chunkLen":256,"inLen":128,"encoding":"utf16le"}: 5.67%
  {"n":400000,"chunkLen":1024,"inLen":128,"encoding":"utf16le"}: 4.26%
  {"n":400000,"chunkLen":16,"inLen":1024,"encoding":"utf16le"}: 1.32%
  {"n":400000,"chunkLen":256,"inLen":1024,"encoding":"utf16le"}: 2.69%
  {"n":400000,"chunkLen":1024,"inLen":1024,"encoding":"utf16le"}: 2.46%

According to nodejs#59186 this
benchmark file takes 6 hours to complete a full benchmark/compare.js
script (60 runs in total) and this regression tests unrealitics to do
between Node.js releases. By using calibrate-n scripts I could find
a better N also ajusting some bench configs. e.g: avoid dead code
elimination by V8.
@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/performance

@nodejs-github-bot nodejs-github-bot added benchmark Issues and PRs related to the benchmark subsystem. string_decoder Issues and PRs related to the string_decoder subsystem. labels Jul 23, 2025
Copy link
Member

@H4ad H4ad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "inLen" of 4096 was added at d23b7d2#diff-95e5dd48819eabf2c86b6b9c2341ea098bcf32dd2c9be1a23d1eea9dded83402R7-R17, I didn't find any reason to keep it, so LGTM.

Also, what's the new time to run this benchmark?

@RafaelGSS
Copy link
Member Author

The "inLen" of 4096 was added at d23b7d2#diff-95e5dd48819eabf2c86b6b9c2341ea098bcf32dd2c9be1a23d1eea9dded83402R7-R17, I didn't find any reason to keep it, so LGTM.

Also, what's the new time to run this benchmark?

From 369063ms (6.15min) to 66000ms (1.06min)

@RafaelGSS RafaelGSS added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. commit-queue Add this label to land a pull request using GitHub Actions. labels Jul 25, 2025
@nodejs-github-bot nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 25, 2025
@nodejs-github-bot nodejs-github-bot merged commit daa9e4b into nodejs:main Jul 25, 2025
34 checks passed
@nodejs-github-bot
Copy link
Collaborator

Landed in daa9e4b

aduh95 pushed a commit that referenced this pull request Jul 25, 2025
According to #59186 this
benchmark file takes 6 hours to complete a full benchmark/compare.js
script (60 runs in total) and this regression tests unrealitics to do
between Node.js releases. By using calibrate-n scripts I could find
a better N also ajusting some bench configs. e.g: avoid dead code
elimination by V8.

PR-URL: #59187
Refs: #59186
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. benchmark Issues and PRs related to the benchmark subsystem. string_decoder Issues and PRs related to the string_decoder subsystem.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants