CI flakiness

[jasnell]

CI appears to be having a number of issues today. Jobs have either been hanging or erroring out in weird ways. I've also attempting to start CI jobs just to have it do absolutely nothing after pressing submit. Can someone give the CI a bit of a kick?

[gibfahn]

Previously discussed in #761 (comment)

[gibfahn]

This seems relevant:

Your Jenkins data directory "/var/lib/jenkins" (AKA JENKINS_HOME) is almost full. You should act on it before it gets completely full.

[rvagg]

Yeah, full disk, working on it, we bumped into not long ago and it turned out to be a couple of very large jobs. We're going to need to go full block device on this machine, perhaps I'll try and do that now eh?

[gibfahn]

We're going to need to go full block device on this machine, perhaps I'll try and do that now eh?

Sounds reasonable, I don't think there was any objection when it was discussed before.

So how does access to the infra machines work? Everyone in build has access to test, the release WG (and some other people?) have access to release, and some subset of people have access to infra?

It'd be great to document this somewhere, maybe the build README.

[refack]

I'm assuming it's probably not the same "disk full issue" but goes under "CI flakiness" title, the node-test-commit-arm-fanned is stuck
https://ci.nodejs.org/computer/node-msft-cross-compiler-1/

[rvagg]

So I think I figured it out, the backup job that also does the cleanup wasn't doing a full cleanup. It uses find to do the job and uses the following roots /var/lib/jenkins/*/builds/ and /var/lib/jenkins/*/configurations/axis-nodes/*/builds/ but the axis jobs have names other than 'axis-nodes' including 'axis-MACHINE', 'axis-label', 'axis-v8test' and a bunch of others. Basically any multi-axis job has a name corresponding to the axis selected. What's more, the build directories are pretty deep within some of these due to the way they are configured.

So, I've changed the find to effectively do this: find /var/lib/jenkins/ -depth -type d -regex '/var/lib/jenkins/.*/builds/[0-9]+' -mtime +7 -exec rm -rf '{}' \;. So it now matches any depth of build directory and finds the job subdirectories underneath that are older than 7 days.

Back to 7 days and we're now at 86% disk usage on the machine by including these extra build directories.

@jbergstroem and @joaocgreis you might want to check my work on that .. particularly the axis thing @joaocgreis since that's your wheelhouse.

Regarding block storage, it turns out the reason we haven't done this yet is that this machine is in DigitalOcean's SFO1 which doesn't do block storage, so we'd have to redeploy the machine in a new datacenter to get this functionality unfortunately.

Regarding access, some of these key machines are accessible only by the "infra" group which is myself, @jbergstroem, @mhdawson and @joaocgreis. ci-release, www, backup(s) and a few others are reserved for this group.

Regarding further flakiness since my last message, as per @refack's comment, that's my fault as I inadvertantly upgraded Jenkins when I shouldn't have and got us into the Java 8 requirement territory as @gibfahn outlined in #775 and a large chunk of the machines couldn't properly connect! I've downgraded it again and it seems to be back to normal but we may need a bit of time to flush out current work.