CI job is broken on Windows

[richardlau]

Looks like the CI job is currently broken on Windows.

e.g. https://ci.nodejs.org/view/post-mortem/job/nodereport-continuous-integration/109/MACHINE=win10/console

[win10] $ sh -xe C:\Users\ADMINI~1\AppData\Local\Temp\hudson4557773619010135318.sh
+ rm -rf 'c:\workspace\nodereport-continuous-integration\MACHINE\win10/*'
+ DOWNLOAD_DIR=https://nodejs.org/download/release/
+ case $NODE_VERSION in
+ DOWNLOAD_DIR=https://nodejs.org/download/nightly/
++ curl https://nodejs.org/download/nightly/
++ grep v8.0.0-nightly
++ sort -t. -k 1,1n -k 2,2n -k 3,3n
++ tail -1
++ cut '-d"' -f 2
++ tr -d /
Input file specified two times.


  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 76806    0 76806    0     0   342k      0 --:--:-- --:--:-- --:--:--  369k
+ LINK=
+ case $MACHINE in
+ OS=win
+ ARCH=x64
+ EXT=zip
+ curl -O https://nodejs.org/download/nightly//node--win-x64.zip

The job is failing to identify the correct version-tag, e.g. v8.0.0-nightly2017030187a039d721, to download.

Googling for the Input file specified two times. suggests that this is because the Windows version of sort is being used rather than the one that comes with the git bash shell. The injectedEnvVars (https://ci.nodejs.org/view/post-mortem/job/nodereport-continuous-integration/109/MACHINE=win10/injectedEnvVars/) seem to back this up:

Name	Value
Path	C:\ProgramData\Oracle\Java\javapath;C:\Python27;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin

I spoke to @gdams who says the sort was added recently so that 6.10.0 comes after 6.9.5 (e.g. on https://nodejs.org/download/release/ v6.10.0/ is listed before v6.2.0/).

cc @nodejs/post-mortem-admins

[richardlau]

hmm for v4 and v6 do we even need to scrape https://nodejs.org/download/release/? Can't we use https://nodejs.org/download/release/latest-v4.x/ and/or https://nodejs.org/download/release/latest-v6.x/?

We'd still need to scrape https://nodejs.org/download/nightly/ for Node.js 8, so still have the sorting problem.

[gdams]

You can do, you lose the ability to run against previous versions though

[richardlau]

Maybe we could scrape https://nodejs.org/download/release/index.json instead? It appears to be sorted within release lines, most recent first.

[richardlau]

https://nodejs.org/download/nightly/index.json for the nightly builds.

[richardlau]

It may actually be easier to scrape the index.tab files instead of index.json.
e.g.

$ NODE_VERSION=v6 && curl -sL https://nodejs.org/download/release/index.tab | awk -v pat=$NODE_VERSION '$1~pat { print $1; exit;}'
v6.10.0
$ NODE_VERSION=v6.9 && curl -sL https://nodejs.org/download/release/index.tab | awk -v pat=$NODE_VERSION '$1~pat { print $1; exit;}'
v6.9.5
$ NODE_VERSION=v8.0.0-nightly && curl -sL https://nodejs.org/download/nightly/index.tab | awk -v pat=$NODE_VERSION '$1~pat { print $1; exit;}'
v8.0.0-nightly20170302172be50fe1

Edit: See nodejs/node#7760 (comment) for something similar.

[rnchamberlain]

1. Forcing the jenkins job to use the cygwin sort, i.e /usr/bin/sort, instead of the Windows sort fixes it for Win 10, see https://ci.nodejs.org/view/post-mortem/job/nodereport-continuous-integration/125/.

2. The unzip was very verbose on windows, fixed that by adding -q option

3. There is still an intermittent failure on win2012: "common.gypi not found". Seems to be machine dependent:
   fails on test-rackspace-win2012r2-x64-7
   https://ci.nodejs.org/view/post-mortem/job/nodereport-continuous-integration/MACHINE=win2012r2/129/console
   passes on test-azure_msft-win2012r2-x64-1
   https://ci.nodejs.org/view/post-mortem/job/nodereport-continuous-integration/MACHINE=win2012r2/128/console

[sxa]

Ref point 3 on the last comment it appears to be referencing c:\workspace\reis-citgm-native-windows-binary within the job. If that workspace isn't on the machine (possible if the job has only ever run on the other box) then that would explain the error.

[rnchamberlain]

@joaocgreis Hi João, re c:\workspace\reis-citgm-native-windows-binary, just wondering if you can shed any light on this problem for us, with your windows expertise? Many thanks!

[sxa]

It's getting into the config.gypi:

find . -type f -print | xargs grep reis-
./node-report/npm-debug.log:33384 verbose unlock done using c:\workspace\reis-citgm-native-windows-binary\smoker\npm-cache\_locks\staging-464173f0da960888.lock for C:\workspace\nodereport-continuous-integration\MACHINE\win2012r2\node-report\node_modules\.staging
./node-report/build/config.gypi:    "nodedir": "c:\\workspace\\reis-citgm-native-windows-binary",
./node-report/build/config.gypi:    "cache": "c:\\workspace\\reis-citgm-native-windows-binary\\smoker\\npm-cache",

[joaocgreis]

@rnchamberlain Hi! That workspace is a left over from a test job I created, it's probably on other machines as well. When I created it, it must have had one run without npm sandboxing and changed the npm userconfig. The best solution is to make sure npm runs in a sandbox in your job, by adding these two lines to the script:

export NPM_CONFIG_USERCONFIG=$WORKSPACE/npmrc
export NPM_CONFIG_CACHE=$WORKSPACE/npm-cache

I added them and started a job with the same parameters as the last one: https://ci.nodejs.org/view/post-mortem/job/nodereport-continuous-integration/130/ , all green on Windows now.

[rnchamberlain]

@joaocgreis perfect, many thanks, closing this now.