pm2 is failing to start after Ubuntu apt-get update/upgrade-s. [PM2] Spawning PM2 daemon with pm2_home

[bluerid]

PM2 is failing to start after an Ubuntu apt-get update/upgrade. It shows this only when started without the debug mode enabled.

[PM2] Spawning PM2 daemon with pm2_home=/home/ubuntu/.pm2

This is hard to replicate as:

I was doing all this in an EC2 instance and after a series of apt-get updates and apt-get upgrade -ys were performed all hell broke loose. Had node v8.0.0 and npm 6.10.0 installed from before while the pm2 was working. If I use the AMI image to spawn a fresh instance PM2 works perfectly there. It's only after using aptitude to update the package lists and install newer versions of the packages using upgrade PM2 fails and shows this error.

$ pm2 --version
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: connect EACCES /home/ubuntu/.pm2/rpc.sock
    at Object.exports._errnoException (util.js:870:11)
    at exports._exceptionWithHostPort (util.js:893:20)
    at PipeConnectWrap.afterConnect [as oncomplete] (net.js:1063:14)

I'm doing the upgrades using a bash script that includes a reboot command at the very end. I can't share the full bash script but here is what's mostly happening within it (just multiple times).

systemctl stop ${SERVICENAME}
sleep 3
wget -qO - https://www.service-x.org/static/pgp/server-4.0.asc | sudo apt-key add -
mv /etc/apt/sources.list.d/service-x-org-3.6.list /etc/apt/sources.list.d/service-x-org-3.6.list.old
sleep 2
echo "deb [ arch=amd64 ] https://repo.service-x.org/apt/ubuntu xenial/service-x-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/service-x-org-4.0.list
apt-get update
apt-get upgrade -y
apt-get upgrade -y service-x-server
sleep 3
systemctl start ${SERVICENAME}
progressbar 2
sleep 3

.......
reboot

After following what was suggested in the thread ($ npm install pm2@latest -g and $ pm2 update) the PM2 now shows this.
[PM2] Spawning PM2 daemon with pm2_home=/home/ubuntu/.pm2

Environment Setup

• Ubuntu 16.04 LTS running on AWS EC2 instance.
• node v8.0.0 and npm 6.10.0
• pm2 2.5.0

Debug Information:

• On a fresh instance:

ubuntu@ip-172-31-1-185:~$ DEBUG=* pm2 list
  pm2:paths pm2 home resolved to /home/ubuntu/.pm2 /home/ubuntu +0ms
  pm2:conf Using 2 parallelism (CONCURRENT_ACTIONS) +8ms
  pm2:client Using RPC file /home/ubuntu/.pm2/rpc.sock +340ms
  pm2:client Using PUB file /home/ubuntu/.pm2/pub.sock +0ms
  pm2:interface:daemon Getting interaction info +1ms
  pm2:interface:daemon [PING INTERACTOR] Trying to connect to Interactor daemon +0ms
  axon:sock connect attempt null:/home/ubuntu/.pm2/interactor.sock +0ms
  pm2:client [PING PM2] Trying to connect to server +5ms
  axon:sock connect attempt null:/home/ubuntu/.pm2/rpc.sock +3ms
  axon:sock error ENOENT +4ms
  axon:sock ignored ENOENT +0ms
  axon:sock connect +0ms
  axon:sock add socket 0 +1ms
  axon:queue flush 0 messages +0ms
  axon:sock closing +1ms
  axon:sock closing 1 connections +0ms
  pm2:client Daemon alive +6ms
  axon:sock remove socket 0 +1ms
  pm2:client Launching RPC client on socket file /home/ubuntu/.pm2/rpc.sock +1ms
  axon:sock connect attempt null:/home/ubuntu/.pm2/rpc.sock +0ms
  axon:sock connect +1ms
  axon:sock add socket 0 +0ms
  axon:queue flush 0 messages +0ms
  pm2:client RPC Connected to Daemon +1ms
  pm2:cli Now connected to daemon +5ms
  pm2:client Calling daemon method pm2:getVersion on rpc socket:/home/ubuntu/.pm2/rpc.sock +0ms
  pm2:client Calling daemon method pm2:getMonitorData on rpc socket:/home/ubuntu/.pm2/rpc.sock +2ms
  pm2:client Calling daemon method pm2:getMonitorData on rpc socket:/home/ubuntu/.pm2/rpc.sock +4ms
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────────┘
 Module activated
┌──────────────────┬─────────┬────────────┬────────┬─────────┬─────┬─────────────┐
│ Module           │ version │ target PID │ status │ restart │ cpu │ memory      │
├──────────────────┼─────────┼────────────┼────────┼─────────┼─────┼─────────────┤
│ pm2-logrotate    │ 2.4.0   │ N/A        │ online │ 0       │ 0%  │ 36.293 MB   │
│ pm2-server-monit │ 2.6.2   │ N/A        │ online │ 0       │ 0%  │ 30.391 MB   │
└──────────────────┴─────────┴────────────┴────────┴─────────┴─────┴─────────────┘
 Use `pm2 show <id|name>` to get more details about an app
  axon:sock closing +23ms
  axon:sock closing 1 connections +0ms
  axon:sock remove socket 0 +0ms
  pm2:client PM2 RPC cleanly closed +13ms
  pm2:cli This command took 0.032s to execute +0ms

• On a updated/upgraded instance:

~/fourthpartner-dashboard-api$ sudo DEBUG=* pm2 start process.json 
  pm2:paths pm2 home resolved to /home/ubuntu/.pm2 /home/ubuntu +0ms
  pm2:conf Using 2 parallelism (CONCURRENT_ACTIONS) +0ms
  pm2:client Using RPC file /home/ubuntu/.pm2/rpc.sock +0ms
  pm2:client Using PUB file /home/ubuntu/.pm2/pub.sock +0ms
  interactor:client [PING INTERACTOR] Trying to connect to Interactor daemon +0ms
  axon:sock connect attempt null:/home/ubuntu/.pm2/interactor.sock +0ms
  pm2:client [PING PM2] Trying to connect to server +5ms
  axon:sock connect attempt null:/home/ubuntu/.pm2/rpc.sock +3ms
  axon:sock error ENOENT +3ms
  axon:sock ignored ENOENT +0ms
  axon:sock error ENOENT +1ms
  axon:sock ignored ENOENT +0ms
  axon:sock attempting reconnect +101ms
  axon:sock closing +1ms
  axon:sock closing 0 connections +0ms
  pm2:client Daemon not launched +106ms
  axon:sock connect attempt null:/home/ubuntu/.pm2/rpc.sock +0ms
  axon:sock attempting reconnect +0ms
  axon:sock closing +1ms
  axon:sock closing 0 connections +0ms
  interactor:client Interactor Daemon not launched +111ms
  axon:sock connect attempt null:/home/ubuntu/.pm2/interactor.sock +0ms
[PM2] Spawning PM2 daemon with pm2_home=/home/ubuntu/.pm2
  axon:sock error ENOENT +31ms
  axon:sock ignored ENOENT +0ms
  axon:sock error ENOENT +0ms
  axon:sock ignored ENOENT +0ms

The debug message is not clear and hard to understand what's causing the problem. Could you please help me figure out what can be causing the problem.

Update:

SyntaxError: Unexpected token ...
    at createScript (vm.js:74:10)
    at Object.runInThisContext (vm.js:116:10)
    at Module._compile (module.js:533:28)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Module.require (module.js:513:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/usr/local/lib/node_modules/pm2/node_modules/picomatch/index.js:3:18)
2019-10-17T04:00:19.090Z pm2:paths pm2 home resolved to /home/ubuntu/.pm2 /home/ubuntu
2019-10-17T04:00:19.093Z pm2:conf Using 2 parallelism (CONCURRENT_ACTIONS)
/usr/local/lib/node_modules/pm2/node_modules/picomatch/lib/picomatch.js:54
    let ignoreOpts = { ...options, ignore: null, onMatch: null, onResult: null };
                       ^^^

SyntaxError: Unexpected token ...
    at createScript (vm.js:74:10)
    at Object.runInThisContext (vm.js:116:10)
    at Module._compile (module.js:533:28)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Module.require (module.js:513:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/usr/local/lib/node_modules/pm2/node_modules/picomatch/index.js:3:18)

This is what I found in the .pm2/pm2.log file.

[chipallen2]

Similar thing happened to me today when starting a fresh pm2 install with latest - 4.0.X. Solution was installing the previous version 3.5.1 with npm i -g pm2@3.5.1 and then it started working again.

I assume root cause was because our server is on Node 6 still and they dropped support in their 4.0 update of PM2 for node 6. Just a hunch - not sure for certain.

[bluerid]

Similar thing happened to me today when starting a fresh pm2 install with latest - 4.0.X. Solution was installing the previous version 3.5.1 with npm i -g pm2@3.5.1 and then it started working again.

I assume root cause was because our server is on Node 6 still and they dropped support in their 4.0 update of PM2 for node 6. Just a hunch - not sure for certain.

Yes, the issue with PM2 daemon failing to spawn surfaced as I updated pm2 to the latest version, after encountering the first problem which is:

$ pm2 --version
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: connect EACCES /home/ubuntu/.pm2/rpc.sock
    at Object.exports._errnoException (util.js:870:11)
    at exports._exceptionWithHostPort (util.js:893:20)
    at PipeConnectWrap.afterConnect [as oncomplete] (net.js:1063:14)

This shouldn't have appeared in the first place. I had to create another instance and then run the script to update. This time there was no error.

[bluerid]

Had to change the permission for /home/ubuntu/.pm2 folder to make it working. Not sure if this is the correct way to go about it, but it is working now.

[Unitech]

Try out https://getpm2.com/

wget -qO- https://getpm2.com/install.sh | bash

This version embeds Node.js 12.x in PM2 4.x (fully transparent install)

And yes PM2 4.x does not support <8.0 Node.js because of dependent modules (chokidar, pslist...)