Changing TestAll.ps1 script to run tests locally with filtering

[guimafelipe]

When working on WinAppSDK, we want to be able to easily run tests to check if our changes are breaking anything or introducing any regressions.

This Pull Request introduces a refactoring on the testing script, moving the core test functions out of TestAll.ps1 to tools\Test.psm1 module, introducing the TestInfo class for holding the test information and refactoring them all into (almost) pure functions for reusability.

Now, the TestAll.ps1 was changed to also work running locally and passing arguments to filter the tests and optionally build them.

A microsoft employee must use /azp run to validate using the pipelines below.

WARNING:
Comments made by azure-pipelines bot maybe inaccurate.
Please see pipeline link to verify that the build is being ran.

For status checks on the main branch, please use TransportPackage-Foundation-PR
(https://microsoft.visualstudio.com/ProjectReunion/_build?definitionId=81063&_a=summary)
and run the build against your PR branch with the default parameters.

[guimafelipe]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[DrusTheAxe]

What's wrong with TestAll.ps1 (in root of the repository)?

If that lacks functionality (e.g. run explicit subset of tests) why not improve TestAll?

Or perhaps fork it to Test.cmd and tools\Test.ps1 and change \TestAll.cmd to run Test.cmd ...options for existing TestAll.cmd behavior...

[guimafelipe]

I used TestAll.ps1 as base to create this script. The idea for this one is to be a tool for developers to quickly iterate on testing based on what I was doing repetitively. I did not want this to merge with what TestAll.ps1 is doing as this later one is used on our build pipelines and requires way more stuff (log files and .wprp for example) than what is intended for a quick local test run to check if some changes broke anything.

My usual loop was to call MSBuild locally to build the tests, and then go to BuildOutput and run the tests. Now thinking back again, I realize it might be better to also include the test building in this script, as it makes sense we always would want to build and test together.

[DrusTheAxe]

I realize it might be better to also include the test building in this script, as it makes sense we always would want to build and test together

I don't know about that. Depends if every test run requires build. I find that's more of a mixed bag. Sometimes I run a test multiple times between changing any code

Also, don't assume dev inner loop doesn't care about log files or .wprp.

Sounds like you're not reinventing TestAll.ps1 so much as BuildAll.ps1

It would be better to enhance those, or perhaps refactor the underlying plumbing from those with lots of options for dev inner-loop use (DoBuild.ps1, DoTest.ps1) and change those to call the new scripts with options to preserve their current behavior

[guimafelipe]

I changed the script to receive more parameters and filtering based on the .testdef properties. Also, made the build step optional.

For making the TestAll.ps1 and the BuildAll.ps1 scripts to share code with this one, it felt it should come in a separate pull request as this one would be way too much. I would like to hear your opinions on this and if it is better to proceed with this one being a separated change after some clean up.

[guimafelipe]

I refactored the test code into a Tests.psm1 module and now TestAll.ps1 is using it

[DrusTheAxe]

I have more feedback but waiting for the TestAll consolidation

[DrusTheAxe]

Can this be a filespec or pattern? e.g. RunTests -TestDef *xyx*.testdef to run all files found recursively matching xyz.testdef

[DrusTheAxe]

P.S. Parameter name suggestions:

• TestDef
• Test
• Path
• File

I think Path most closely matches common Powershell use

[guimafelipe]

I think we can change this to be a pattern. I first went for simplicity following the loop I had with testing locally, and usually I wanted to test everything under a .testdef file to check for regressions or tests breaking. But sometimes we want to test specific tests, so a parameter would be good to have and could override the parameters defined in the .testdef file.

[DrusTheAxe]

-Path ==> Get-ChildItem -Recurse $path

-Blah ==> For any .testdef found via -Path only run it if the blah property in the .testdef matches
there's no 'name' property in a testdef but could do this for the meaningful properties as a regex value e.g.

-Property:Description "Package.*Test.Add" would only run this one case in test\packagemanagerapi\test.testdef

        {
            "Description": "PackageManagerTests.Add* tests for feature PackageManager (arm64 not currently enabled)",
            "Filename": "PackageManagerTests.dll",
            "Parameters": "/name=Test::PackageManager::Tests::PackageDeploymentManagerTests_Add*",
            "Architectures": ["x86", "x64"],
            "Status": "Enabled"
        },

as in if ($PropertyDescription -match $DescriptionFromParsedTestDefJson)

[DrusTheAxe]

BTW take a look at test\packagemanager\api\Test.testdef for an example of a .testdef with multiple Tests. One should be able to easily run a subset of Tests in a .testdef

[guimafelipe]

I think the current version of the script supports these scenarios.

[guimafelipe]

Removed the RunTests.ps1 script completely and refactored TestAll.ps1

[DrusTheAxe]

I realize it might be better to also include the test building in this script, as it makes sense we always would want to build and test together

I don't know about that. Depends if every test run requires build. I find that's more of a mixed bag. Sometimes I run a test multiple times between changing any code

Also, don't assume dev inner loop doesn't care about log files or .wprp.

Sounds like you're not reinventing TestAll.ps1 so much as BuildAll.ps1

It would be better to enhance those, or perhaps refactor the underlying plumbing from those with lots of options for dev inner-loop use (DoBuild.ps1, DoTest.ps1) and change those to call the new scripts with options to preserve their current behavior

[DrusTheAxe]

-Path ==> Get-ChildItem -Recurse $path

-Blah ==> For any .testdef found via -Path only run it if the blah property in the .testdef matches
there's no 'name' property in a testdef but could do this for the meaningful properties as a regex value e.g.

-Property:Description "Package.*Test.Add" would only run this one case in test\packagemanagerapi\test.testdef

        {
            "Description": "PackageManagerTests.Add* tests for feature PackageManager (arm64 not currently enabled)",
            "Filename": "PackageManagerTests.dll",
            "Parameters": "/name=Test::PackageManager::Tests::PackageDeploymentManagerTests_Add*",
            "Architectures": ["x86", "x64"],
            "Status": "Enabled"
        },

as in if ($PropertyDescription -match $DescriptionFromParsedTestDefJson)

[DrusTheAxe]

BTW take a look at test\packagemanager\api\Test.testdef for an example of a .testdef with multiple Tests. One should be able to easily run a subset of Tests in a .testdef