CitGM jobs don't end on AIX

[Trott]

I took test-osuosl-aix61-ppc64_be-2 offline in Jenkins (so I guess that's "offline" rather than offline) as it was failing repeatedly with:

Started by upstream project "node-test-commit-aix" build number 8918
originally caused by:
 Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on test-osuosl-aix61-ppc64_be-2 (aix61-ppc64) in workspace /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
Cloning the remote Git repository
Cloning repository https://github.com/nodejs/node.git
 > git init /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64 # timeout=30
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Could not init /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:717)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:511)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
	at hudson.remoting.UserRequest.perform(UserRequest.java:152)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:332)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
	at hudson.remoting.Engine$1$1.run(Engine.java:85)
	at java.lang.Thread.run(Thread.java:809)
	at ......remote call to Channel to /140.211.9.100(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
	at hudson.remoting.Channel.call(Channel.java:830)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
	at sun.reflect.GeneratedMethodAccessor539.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
	at com.sun.proxy.$Proxy79.execute(Unknown Source)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1075)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
	at hudson.scm.SCM.checkout(SCM.java:496)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1281)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
	at hudson.model.Run.execute(Run.java:1728)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:405)
Caused by: hudson.plugins.git.GitException: Error performing command: git init /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1931)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1892)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1888)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1533)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:715)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:511)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
	at hudson.remoting.UserRequest.perform(UserRequest.java:152)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:332)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
	at hudson.remoting.Engine$1$1.run(Engine.java:85)
	at java.lang.Thread.run(Thread.java:809)
Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64"): error=11, Resource temporarily unavailable
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1059)
	at hudson.Proc$LocalProc.<init>(Proc.java:245)
	at hudson.Proc$LocalProc.<init>(Proc.java:214)
	at hudson.Launcher$LocalLauncher.launch(Launcher.java:846)
	at hudson.Launcher$ProcStarter.start(Launcher.java:384)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1920)
	... 16 more
Caused by: java.io.IOException: error=11, Resource temporarily unavailable
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:212)
	at java.lang.ProcessImpl.start(ProcessImpl.java:164)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1040)
	... 21 more
ERROR: Error cloning remote repo 'origin'
Run condition [Always] enabling perform for step [[]]
TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: *.tap
Did not find any matching files. Setting build result to FAILURE.
Checking ^not ok
Jenkins Text Finder: File set '*.tap' is empty
Sending e-mails to: michael_dawson@ca.ibm.com gib@uk.ibm.com Bethany.Griggs@uk.ibm.com
Notifying upstream projects of job completion
Finished: FAILURE

ping @mhdawson

[gibfahn]

ps -ef | grep citgm | awk '{print $2}' | xargs kill

seems to have fixed it. We should look at adding cleanup to the citgm jobs.

[gibfahn]

@Trott btw it's worth cc'ing @nodejs/platform-aix

[gibfahn]

First build that ran on it seems to be going okay, so closing.

https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/8923/console

[refack]

BTW: @gibfahn did you clean the ramdrive? I think the jobs go zombie when the ramdrive if full.

[gibfahn]

@refack it was only 10% full.

[refack]

@refack it was only 10% full.

Puff goes my hypothesis 🤔

[refack]

So how about adding ps -ef | grep citgm | awk '{print $2}' | xargs kill unconditionally to the end of the job?

[gibfahn]

So how about adding ps -ef | grep citgm | awk '{print $2}' | xargs kill unconditionally to the end of the job?

SGTM

[refack]

@Trott and I just found 88 unterminated node processes on test-osuosl-aix61-ppc64_be-1
I'm reopening, since we need to either kill the processes on cigtm job end, or take the AIXes out of the citgm roster.