SLOW OR FAILED (500 ERROR) NODE.JS DOWNLOADS

[watson]

^{Edited by the Node.js Website Team}

⚠️ PLEASE AVOID CREATING DUPLICATED ISSUES

Learn more about this incident at https://nodejs.org/en/blog/announcements/node-js-march-17-incident

tl;dr: The Node.js website team is aware of ongoing issues with intermittent download instability.

More Details: #1993 (comment)

_{Original Issue Below}

---

All binary downloads from nodejs.org are very slow currently. E.g. https://nodejs.org/dist/v13.0.1/node-v13.0.1-darwin-x64.tar.xz.

In many cases, they will time out, which affects CI systems like Travis who rely on nvm to install Node.js.

According to @targos, the CPU of the server is spinning 100% being used by nginx, but we can't figure out what's wrong.

[targos]

Top of the top:

[targos]

@nodejs/build-infra

[atulmy]

Same here, using tj/n, the download just terminates after a while

[kachkaev]

Azure DevOps pipelines also suffer, that's because a standard Node.js setup task downloads the binary from nodejs.org.

Kudos to those who are looking into the CPU usage ✌️ You'll nail it! 🙌

[sam-github]

@targos what machine is the top output from?

Fwiw, only infra team members are: https://github.com/orgs/nodejs/teams/build-infra/members and Gibson isn't current, he should be removed.

[MylesBorins]

I wonder what traffic looks like... could be a DOS

We've talked about it in the past, but we really should be serving these resources via a CDN... I believe that the issue was maintaining our metrics. I can try and find some time to help out with the infrastructure here if others don't have the time

[sam-github]

It could also be that 13.x is super popular? Hard to know without access to access logs.

@MylesBorins the infrasture team is even less staffed than the build team as a whole, if you wanted to pick that up, it would be 🎉 fabulous.

[mhdawson]

The issue has been the metrics in terms of why they are not on a CDN, @joaocgreis and @rvagg do have an active discussion on what we might do on that front.

@MylesBorins if you can get help for this specific issue that might be good (please check with @rvagg and @joaocgreis ). Better would be ongoing sustained engagement and contribution to the overall build work.

[targos]

@sam-github it's from direct.nodejs.org. I don't have root access to the machine, so top is basically the only thing I could do to help.

[sam-github]

I don't even have non-root access, I guess you do because you are on the release team.

We'll have to wait for an infra person.

[mhdawson]

I can help take a look after the TSC meeting.

[michaelsbradleyjr]

In addition to downloads only intermittently succeeding, when they do succeed it seems that sometimes (?) the bundled node-gyp is broken. I'm seeing that behavior in CI builds on Azure Pipelines.

For linux builds with successful downloads of Node.js v8.x and v10.x, node-gyp fails to build sources that were previously unproblematic.

For windows builds with successful downloads of Node.js v8.x and v10.x, I'm seeing errors like this:

gyp ERR! stack Error: win-x86/node.lib local checksum 629a559d347863902b81fad9b09ba12d08d939d20a0762578e904c6ee1a7adab not match remote fea7d0c5a94fc834a78b44b7da206f0e60ef11dd20d84dc0d49d57ee77e20e16

That is, in the case of windows node-gyp doesn't attempt to compile; rather, it fails outright owing to the checksum error.