Question: how detect and deal with zombie embedded processes

[ericedem]

Hello, and thanks for the awesome library! We are using embedded postgres heavily in our tests which has been great.

One problem we are having is that when the process dies unexpectedly we end up with zombie processes because we were unable to call Stop() on the postgres db. In general we are able to catch this due to expected failures like failed tests, but sometimes we can't like when a debugger is running the process and gets terminated instead of running to completion.

Is there a reasonable way people have come up to deal with this problem? The best ideas I have had is using a terminal command like netstat to check a known port at test startup and try to kill any zombies but this is not ideal as it exits the test runtime and could interfere with normal processes on the host machine.

Cheers.

[fergusstrange]

Hi @ericedem,

Thanks for the question. This is definitely a problem we've seen a lot and it's a really hard one to solve. We have put in place checks to ensure that there's some level of helpful messaging when an existing process is present but we really have tried to steer away from automatically killing things out of an abundance of safety.

The best approach I've seen is to use a build system that runs in a containerised enviornment, with the whole container shut down at the end of the tests ensuring that child processes are killed too. Our very own tests run using this approach.

Another good approach if you're running multiple tests is to ensure the port number is increased between each test. This will ensure you shouldn't get any clashes for a given test suite.

Let me know if any of this helps or you have any other good ideas!

[ngotchac]

One trick we are using, which for now has been quite effective, is to run the binaries through this bash script:

#!/usr/bin/env bash

set -euo pipefail

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
CUR_PID="$$"
PAR_PID=$(ps -o ppid= -p $CUR_PID)


# Start the binary in the background
mybin &
# Store the PID of the process
MY_PID=$!

function cleanup() {
    if ps -p $MY_PID > /dev/null
    then
        kill -SIGTERM $MY_PID
    fi
    set -e
}

# Kill all subprocess on exit/error
trap cleanup SIGINT SIGTERM EXIT

printf "Running [CUR_PID=$CUR_PID] [PAR_PID=$PAR_PID] [MY_PID=$ANVILMY_PID_PID]\n"

while true; do
    # Ensures that it is still running (could have been stopped, or errored)
    if [ -z "$(ps -o ppid= -p $MY_PID)" ]
    then
        exit 1
    fi
    # Ensure the parent process that executed this script still exists
    if [ -z "$(ps -o ppid= -p $PAR_PID)" ]
    then
        exit 0
    fi
    sleep 1
done

which starts a process in the background, and exits when the parent process doesn't exist anymore.

Would you be open to use this kind of setup?