Additional builders: shared libraries and debug

[jbergstroem]

I'd like to improve testing against shared libraries as well as a debug build. The debug-build is a no-brainer (./configure --debug), but the shared build would introduce a bit of flavour.

Since quite a few iojs/nodejs packagers (linux distributions, freebsd, pkgsrc) uses at least a shared zlib and openssl the optimal scenario would be being able to use a package manager to maintain these dependencies instead of building them ahead of every jenkins run. My vision is to – as closely as possible – represent how it's actually used.

My plan is to use archlinux as a base os since it has most of our dependencies both packaged and up to date. If anyone has input on a more suiting distribution, I'm all ears. Since I use Gentoo myself I'm just going to exclude it on the grounds of having enough coverage already (it's a pretty good distribution for working with multiple toolchains and dependencies).

Thoughts?

[bnoordhuis]

Just curious, how are you going to deal with floating patches?

[jbergstroem]

@bnoordhuis my view is that we should respect abi and not apply floating patches without a high amount of resistance. I for instance PR'ed removing building against a shared c-ares for this reason, but optimally we'd upstream patches and let them flow through there before patching them locally. I wouldn't really see this as a build problem, rather a problem with how the iojs WG/TC handles patches in deps/.

[bnoordhuis]

Most floating patches aren't ABI breaks, they're bug fixes. c-ares is an exception because upstream is no longer actively developed.

[jbergstroem]

Regardless, the bug fixes should go upstream no? Lets run through them:

• libuv: you and saghul. I'll assume that whatever we patch we want upstream
• http_parser: indutny and more. Same thing?
• openssl: I doubt we'll accept floating patches here that will affect shared vs bundled (for instance the cli patch that we might land to improve windows buildbot status).
• v8: we might float for a version, but everything we've done so far afaik is upstreamed.

I think we should let whatever bug we fix in tree fail until it lands in upstream. With that said, having it part of the incremental build might not be what we want. We could definitely start by having it part of iojs+any-pr+multi until we end up getting discrepancies.

[bnoordhuis]

I don't think it's that clear cut. To take that OpenSSL CLI patch as an example, it fixes an issue for us but it's not the kind of patch that upstream will gladly (or ever) take. Now what?

Also, I think you're overlooking the LTS aspect. If we commit to longer-term support, we effectively become the maintainers of dependencies with a shorter support cycle, like V8. We'll be carrying an ever increasing number of patches.

I think we should let whatever bug we fix in tree fail until it lands in upstream.

You mean wait until an upgrade pulls in the bug fix instead of applying it ahead of time? I speculate you're not going to find much support for a proposal like that, people like their bug fixes.

[mhdawson]

I agree that we should aim to have all floating patches upstreamed, but particularly in the LTS releases that won't always be possible. V8 is a good example were we can only contribute to bleeding edge, and beta/stable, and in terms of stable for example it would likely have to be relatively important to get it. So we'll have to handle the case were we need to float a patch on the V8 version that is in the LTS stream while at the same time making sure that we contributed it back so that we don't have to float in later versions.

[jbergstroem]

@bnoordhuis
I don't think it's that clear cut. To take that OpenSSL CLI patch as an example, it fixes an issue for us but it's not the kind of patch that upstream will gladly (or ever) take. Now what?

It only went in because it didn't change the api or abi, just how we use the openssl cli to test on windows. I'm not saying that we should see this as a case-by-case, but I don't really see a scenario where we'd allow floating patches on top of things like zlib or openssl. If we look at what's been going on in deps/, we rarely touch anything but npm or v8 between upstream releases.

@bnoordhuis
You mean wait until an upgrade pulls in the bug fix instead of applying it ahead of time? I speculate you're not going to find much support for a proposal like that, people like their bug fixes.

Yeah – you're right. That's why I also think that down the road (or perhaps earlier) a shared builder is more about release conformance than testing every revision/branch. Perhaps toss it in when rc's are cut? Part of what I'm trying to accomplish is to let all packagers know we care about how they've chosen to use our configure options.

@bnoordhuis
Also, I think you're overlooking the LTS aspect.

@mhdawson
V8 is a good example were we can only contribute to bleeding edge, and beta/stable, and in terms of stable for example it would likely have to be relatively important to get it.

I agree. V8 is also one of the dependencies we don't allow building against a shared library because of its unpredictable development.

Just to be clear here: I'm talking about a shared openssl, zlib, libuv, http-parser and – as of recent – icu. I did a survey some months ago and found that pretty much all third party packagers (os distributions, package managers, ..) uses a shared zlib and most also a shared openssl. After that it diminishes somewhat, but there are occurrences of libuv. Actually, npm is also a common decouple.

[jbergstroem]

I'm starting to feel that we need better coverage on --debug than just another bot. It currently fails to build on SmartOS. Perhaps another job that runs debug on all slaves every now and then?

[bnoordhuis]

It's not a bad idea but there are probably going to be a lot of spurious failures because debug builds are so slow. I guesstimate that a run on the rpi 1 will take three or four hours, maybe more.

[jbergstroem]

@bnoordhuis yeah, it's brutal. We're unavoidably moving towards more build groups anyway (as in, different jobs for different type of problems; PR, pre-release, etc) -- so having debug part of something we run occasionally is most likely what we want.

[jbergstroem]

Update: we now have a debug builder: https://ci.nodejs.org/job/jbergstroem-test-commit-debug/

This will be tested for a while and then integrated as a part of a recurring test runner. I aim to add freebsd and smartos vm's in there as well.

[orangemocha]

We could also add a parameter to node-test-commit.

[rvagg]

Came up recently @ nodejs/node#16432, http2 debug builds not compiling at all and we didn't notice until after it was pushed out in a release

[rvagg]

Experimenting with shared library linking @ https://ci.nodejs.org/job/node-test-commit-linux-linking/nodes=ubuntu1604_openssl110_x64/3/ for testing the upcoming OpenSSL 1.1 support @ nodejs/node#16130

Extends #992 and adds a Docker container that downloads and compiles openssl 1.1.0g and puts it in /opt/openssl. Then in Jenkins it does like the FIPS job and does a custom ./configure to link followed by make test-ci.

I'm thinking that we could expand this and put a bunch of shared libraries in /opt/ and have a pool of these containers that we pull in and run them all as part of the normal build & test process. Add in zlib and cares and we'd have good coverage.