Test suite running only partially since PHPUnit 9.1.3 (?) when PHPUnit called with file name to test suite setup file

[jrfnl]

Q	A
PHPUnit version	9.1.3 - 9.3.10
PHP version	7.3 + 7.4
Installation Method	Composer / PHAR - both

Summary

This is a bit of an awkward one to explain, so please bear with me. I will also include a solution which solved the issue for us.

With that in mind, feel free to close this issue if you deem this an acceptable BC-break, but I figured it would be helpful to document the issue anyway in case other people run into the same thing.

TL;DR: Calling PHPUnit with a file name pointing to a file which sets up a test suite, doesn't run the complete test suite anymore, but only part of it.

I suspect the issue is a regression related to issue #4232

Current behavior

Given the following config:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit backupGlobals="true" beStrictAboutTestsThatDoNotTestAnything="false" bootstrap="tests/bootstrap.php">
    <testsuites>
        <testsuite name="PHP_CodeSniffer Test Suite">
            <file>tests/AllTests.php</file>
        </testsuite>
    </testsuites>
</phpunit>

Where the tests/AllTests.php file contains a TestSuite setup along the following lines:

namespace PHP_CodeSniffer\Tests;

class PHP_CodeSniffer_AllTests
{
    public static function suite()
    {
        $suite = new TestSuite('PHP CodeSniffer');
        $suite->addTest(Core\AllTests::suite());
        $suite->addTest(Standards\AllSniffs::suite());
        return $suite;
    }
}

And running PHPUnit with the following command:

phpunit ./tests/AllTests.php

Only 368 tests are run instead of the expected 567 tests.

Basically, the tests added via $suite->addTest(Core\AllTests::suite()); are run and those added via $suite->addTest(Standards\AllSniffs::suite()); are not, or rather are not even added.

(and please don't look at me for how the rest of the test suite is set up... not my code .. legacy project ... etc etc)

How to reproduce

1. Clone the PHP_CodeSniffer repo - git clone git@github.com:squizlabs/PHP_CodeSniffer.git
2. Switch to the 4.0 branch - git checkout 4.0
3. Using PHPUnit 9.1.2, run phpunit tests/AllTests.php and see a test count at the bottom of 567 tests having run.
4. Using a later PHPUnit release, run phpunit tests/AllTests.php and see a test count of 368 tests having run.

The solution I found to fix this was to run the tests without the filename, so just running them with a plain phpunit call.

So:
5. Run the following on any PHPUnit version to see the correct number of 567 tests being run: phpunit (without the tests/AllTests.php)

The fact the file name was still part of the default command is a project legacy from a time it was necessary, just don't ask me when that was 😉

Expected behavior

The test count to be the same cross-version.

External references:

• Issue with more details about the versions I tested the issue on while debugging in the PHP_CodeSniffer repo: PHPCS 4.x | Test suite broken squizlabs/PHP_CodeSniffer#3120

[sebastianbergmann]

Where the tests/AllTests.php file contains a TestSuite setup along the following lines:

Let me stop you right there: The "AllTests.php" approach of manually/programmatically creating a TestSuite object has not been supported for years. @flow-control did a lot of work in the recent past to remove remnants of this functionality from the codebase.

[sebastianbergmann]

feel free to close this issue if you deem this an acceptable BC-break

Done.

[jrfnl]

The "AllTests.php" approach of manually/programmatically creating a TestSuite object has not been supported for years.

Glad to hear it as I personally found it quite unintuitive to work with.
Is there any publication or changelog available in which the dropping of the support way back when is documented which I can refer the maintainer of this particular test suite to ?

feel free to close this issue if you deem this an acceptable BC-break

Done.

👍 More than anything, I opened the issue to allow for other people searching for a solution to find one.

[sebastianbergmann]

This was during the PHPUnit 3 days, before we had a proper changelog. Sorry.

[jrfnl]

@sebastianbergmann Ok, that happens. Thanks for responding.