coverage.py isn't showing the expected results for celery prefork model

[BPraveenRaja]

Describe the bug
coverage.py isn't working for the processes that are not spawned, celery uses prefork model to create processes

To Reproduce
How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

1. What version of Python are you using?
   Python3.9
2. What version of coverage.py shows the problem? The output of coverage debug sys is helpful.
   the latest version
3. What versions of what packages do you have installed? The output of pip freeze is helpful.
   amqp==5.1.1 billiard==4.1.0 blinker==1.6.2 celery==5.3.1 click==8.1.7 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.3.0 coverage==7.3.0 Flask==2.3.3 itsdangerous==2.1.2 Jinja2==3.1.2 kombu==5.3.1 MarkupSafe==2.1.3 prompt-toolkit==3.0.39 python-dateutil==2.8.2 redis==5.0.0 six==1.16.0 tzdata==2023.3 vine==5.0.0 wcwidth==0.2.6 Werkzeug==2.3.7
4. What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix.
   Explained the problem in the next message.
5. What commands should we run to reproduce the problem? Be specific. Include everything, even git clone, pip install, and so on. Explain like we're five!
   Explained the problem in the next message.

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

[nedbat]

Hi, can you provide us with code to run to demonstrate the problem? It's hard to debug your situation without it.

[BPraveenRaja]

Hi, I'm sorry for the late reply. I lost track of this issue.

Here is the link to the code repo url

This application uses flask and celery. This application generally creates a file using celery, and it uses Flask web framework

Please note I'm using sitecustomize.py and COVERAGE_PROCESS_START environment variable to start the coverage process.

Scenario 1: When I ran the celery with pool solo | threads
In one terminal, I ran python run.py
and in another terminal, I ran celery -A celery_worker.celeryInstance worker --loglevel=debug --pool=solo

I executed a few cases in the browser, I have hit the URL in the browser to create a file using Flask and celery
The coverage report is perfectly shown below

Scenario 2: When I ran the celery with pool prefork(default)
In one terminal, I ran python run.py
and in another terminal, I ran celery -A celery_worker.celeryInstance worker --loglevel=debug --pool=prefork

I executed the same cases as scenario 1 in the browser, I hit the URL in the browser to create a file using Flask and Celery
The coverage report is wrongly shown below

[sprezz-arthur]

If I may hijack this issue, it seems the problem comes from:

• Celery uses it's own Pool implementation with billiard.Pool when running under prefork, and it doesn't play with coverage at all
• Celery solo pool, is not really a Pool, and it's just running code synchronously, so it interacts properly with coverage (but defeats the purpose of multiprocessing)

Ultimately this means that coverage does not provide any statistics on code run through Celery Tasks (at least when they run under the usual pool=prefork).

I added some example code https://github.com/sprezz-arthur/flaky-coverage-mre based on my previous issue #1910

You can run it simply with:

python -m cov_stats get_cov_partials.sh 1

[BPraveenRaja]

Thanks. I did make it work with gevent. I'm not sure why it works for that case but not others.

[sprezz-arthur]

Thanks. I did make it work with gevent. I'm not sure why it works for that case but not others.

Basically because using anything other than prefork you're not using billiard.Pool, which seems to be the sole thing that does not play well with coverage.

[nedbat]

Could we make a simple example that uses billiard so we can debug the problem?

[arthurnalves]

If I may hijack this issue, it seems the problem comes from:

* Celery uses it's own Pool implementation with `billiard.Pool` when running under `prefork`, and it doesn't play with `coverage` at all

* Celery `solo` pool, is not really a Pool, and it's just running code synchronously, so it interacts properly with `coverage` (but defeats the purpose of multiprocessing)

Ultimately this means that coverage does not provide any statistics on code run through Celery Tasks (at least when they run under the usual pool=prefork).

I added some example code https://github.com/sprezz-arthur/flaky-coverage-mre based on my previous issue #1910

You can run it simply with:

python -m cov_stats get_cov_partials.sh 1

@nedbat
I've provided an example based on our previous discussion. It's as simple as using billiard.Pool to run a function.
I'd also like to point out the previous suggestion (explicitly closing and joining pool within the context-manager) doesn't change anything here.

[nedbat]

Thanks, sorry I missed that. Can you include instructions for how to properly install that repo?

[arthurnalves]

Thanks, sorry I missed that. Can you include instructions for how to properly install that repo?

Away from keyboard at the moment. I will provide a complete setup when I get back tomorrow.
For the time being I believe any version of python with:

• celery (which includes billiard)
• coverage

Will do.

[nedbat]

I got it to run with poetry install.

I've written an issue against billiard: celery/billiard#419