Add regex "unit tests" test project

[stephentoub]

This follows the convention used by the networking tests, which typically have two distinct test projects per library: functional tests and unit tests. The former are what we typically refer to as our tests for a library, whereas the latter build product source into the test project in order to directly validate internals (an alternative to this is to use InternalsVisibleTo, but that negatively impacts trimming and makes it more challenging to maintain a property boundary for the functional tests). All the existing tests are moved unedited into the FunctionalTests, and a new UnitTests project is added with some initial tests for the RegexTreeAnalyzer and RegexFindOptimizations code; we can expand them further subsequently. The generator parser tests were also consolidated into the functional tests, as there's no longer a good reason for those few tests to be separate.

I fixed a few bugs in RegexTreeAnalyzer as a result. In particular, we were over-annotating things as potentially containing captures or backtracking because in the implementation we were using the lookup APIs meant to be used only once all analysis was complete. This doesn't have a negative functional impact, but it does negatively impact perf of compiled / source generator, which then generate unnecessary code. We were also incorrectly conflating atomicity conferred by a grandparent with atomicity conferred by a parent; we need MayBacktracks to reflect only the atomicity directly contributed by a node, not by its parent's influence, as we need the parent to be able to understand whether the child might backtrack.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This follows the convention used by the networking tests, which typically have two distinct test projects per library: functional tests and unit tests. The former are what we typically refer to as our tests for a library, whereas the latter build product source into the test project in order to directly validate internals (an alternative to this is to use InternalsVisibleTo, but that negatively impacts trimming and makes it more challenging to maintain a property boundary for the functional tests). All the existing tests are moved unedited into the FunctionalTests, and a new UnitTests project is added with some initial tests for the RegexTreeAnalyzer and RegexFindOptimizations code; we can expand them further subsequently. The generator parser tests were also consolidated into the functional tests, as there's no longer a good reason for those few tests to be separate.

I fixed a few bugs in RegexTreeAnalyzer as a result. In particular, we were over-annotating things as potentially containing captures or backtracking because in the implementation we were using the lookup APIs meant to be used only once all analysis was complete. This doesn't have a negative functional impact, but it does negatively impact perf of compiled / source generator, which then generate unnecessary code. We were also incorrectly conflating atomicity conferred by a grandparent with atomicity conferred by a parent; we need MayBacktracks to reflect only the atomicity directly contributed by a node, not by its parent's influence, as we need the parent to be able to understand whether the child might backtrack.

Author:	stephentoub
Assignees:	stephentoub
Labels:	area-System.Text.RegularExpressions
Milestone:	-

[joperezr]

public class RegexRunnerTests

You don't have to do this in your PR, but When creating these I did feel like they were more unit-test type of tests as opposed to the rest of our functional tests. I can, in a follow up PR, move these to the UnitTest project and use RegexRunner directly as opposed to using reflection to get fields/properties/methods

[elinor-fung]

new UnitTests project is added with some initial tests for the RegexTreeAnalyzer and RegexFindOptimizations code

Are these intended to be running in PR right now? The libraries test build just looks for *.Tests.csproj, not *.UnitTests.csproj, so we won't build/run for the current name of System.Text.RegularExpressions.UnitTests.csproj.

runtime/src/libraries/tests.proj

Line 371 in d9eafd0

<ProjectReference Include="$(MSBuildThisFileDirectory)*\tests\**\*.Tests.csproj"

Other existing unit tests do *.Unit.Tests.csproj (under a UnitTests folder).

[stephentoub]

The libraries test build just looks for *.Tests.csproj, not *.UnitTests.csproj

Well that's annoying :-)

Thanks for noticing. Will fix.

[joperezr]

Oh I missed that! I would’ve sworn it was doing *Tests.csproj