Refactor how tests are run

[sebastianbergmann]

• Move test execution logic out of TestResult
  • Use TestResult just as a Collecting Parameter to collect test results
• Before the test execution starts
  • Use a TestSuiteIterator to get all tests that are to be run from the top-level TestSuite
    • Use FilterIterator implementations to filter tests based on --filter, --group, and --exclude-group, for instance
    • This allows the number of tests that are to be run to be calculated correctly before the tests are actually run
  • Sort the tests according to @depends annotations
• Delay creation of TestCase objects and clean them up immediately after a test has been run
  • Allows for more elegant implementation of test execution in isolated PHP processes
  • Allows for running the tests in random order
  • Reduces memory footprint

[dharkness]

I'm very excited to see this! We've had to increase our memory limit every so often to deal with test cases sticking around to the end. One thing you might not have considered yet is that by default any exceptions thrown during a test will have a reference to the test case itself in the backtrace.

As I've started working on a new Java project at my job I'm picking up the new JUnit features. One neat feature is the ability to add a TestDecorator to any test case using an annotation. This is how jMock checks its mocks at the end of each method, and a similar docblock annotation could be helpful for PHP tool integrators.

[banks]

Sebastian, this looks like a good move. It also looks like it will resolve the issues I reported in #261.

Can you confirm that the behaviour I mention there is not intentional and won't be somehow preserved by your refactor?

[elblinkin]

I just did a quick hack where I have a single TestResult and I use it to execute on a loop of TestSuite-s which each correspond to a single file, so that I can deallocate the suite after each test file is executed. I saw in my test suite a reduction from 643MB to 431MB. So very excited to see this implemented better.

[Alexei-Kornienko]

Hello @sebastianbergmann could you please add more details to this issue?
It would be nice if you could create some kind of a document with list of features/requirements that are planned for 4.0 release for example:

• long_class_names -> namespaces
• parallel test execution
• etc., etc.

Basically I think that new 4.0 branch is a good moment to solve old problems and implement existing functionality in cleaner way and I hope I'll be able to help in coding it.