Test failures in WPF

[rchauhan18]

Description

I am a member of the WPF team, and we have recently encountered an issue with two of our DRTs (Test) failing in the WPF. These tests were passing successfully in .NET 8 preview 2, but they started failing after upgrading to .NET 8 preview 3.
The specific area of concern is that we haven't made any changes in the codebase or functionality in area for which these tests belong.
We would greatly appreciate assistance in understanding and resolving this issue. Our goal is to identify the root cause of the test failures in the DRTs and we think that it could be related to runtime.

As part of our investigation, we performed an experiment by replacing the latest WPF binaries in .NET 8-preview2 and running our tests. Both tests passed successfully. We then proceeded to replace the Runtime binaries from .NET 8-preview2 in preview 5, and in this case too both tests passed. These results led us to suspect that the changes causing the failures might have been introduced in the runtime after the preview 2 release.

If we run the exe directly, we can see the test runs properly (without any failure) which shows that there is no problem in out test code. Apart from this we haven't made any changes in the test infrastructure, so it could not be the reason of test failure as the test was passing earlier (before preview3). So this eliminates our test and our infrastructure code.

Tests:

DRT Window Scroll:
In this DRT, we focus on validating the behavior of the ScrollViewer control within the context of a Page's style.
The test scenario involves the following application structure:
Window -> Page -> ScrollViewer -> Rectangle (figure)
To accomplish this, we attach an event handler (OnLoaded) to the Page. When the Page is loaded, this event handler triggers multiple tests to verify the behavior of the content presenter and the visibility of the scroll bar. These tests are conducted with different content sizes, ensuring comprehensive coverage.

DRT Window Functionality:
We aim to validate different functionalities and behaviors of the Window control and test that setting visibility is async.
In this test, we have verified the size and visibility of content in various situation where we changed the size, position and state of the window (Normal, minimized, maximized).

Reproduction Steps

rchauhan18/DrtTests (github.com)

1. Clone the repo
2. Move to the folder RuntimeTest/DrtTests
3. Run command '.\Build.cmd'
4. Move to the test folder 'cd .\publish\Debug\x86\Test'
5. Run the test

.\RunDrts.cmd /name=[NAME]
[NAME] : drtwindowscroll or drtwindowfunctionality

Expected behavior

The test should pass.

Actual behavior

DRT Window Scroll
The test is failing when we are checking the child of Page after loading. The error is 'Specified index is out of range or child is null'. This might have happened due to the onloaded function is called before the page is completely loaded.

DRT Window Functionality
The test is failing because the order of event firing is wrong. The OnContentRendered is called before OnActivation.

Regression?

Yes

Known Workarounds

If we change runtime version to .NET 8.0-preview2, the test will pass.

Configuration

.NET version 8.0-preview3
OS: Win11 22H2

Other information

No response

[jeffschwMSFT]

Can you share a callstack of the failure? Nearly all the time, an ArgumentOutOfRangeException is due to a bug in user code (the app). The issue could even be latent and show up due to timing.

[rchauhan18]

Hey @jeffschwMSFT, thanks for the reply.
Callstack is attached below.
We ran latest wpf code with the runtime version 8.0-preview2 and the test was running as expected. So, this confirms that there is no issue in wpf or the test application code.
CallStack.txt

[jeffschwMSFT]

Given all the code is WPF, what was your analysis on why the callbacks are firing in different orders? Please do first tier investigation and then we can take a look. If you need a hand, we can assist you in your investigation.

[jeffschwMSFT]

My recommendation on the first step, is to create a reduced repro. The minimal set of steps that still repro the issue.

[jeffschwMSFT]

transferring to wpf, while initial investigation is happening

[rchauhan18]

During WPF test execution, the driver that launches the child processes is configured via ProcessStartInfo with UseShellExecute = False and WindowStyle = Hidden.

After the below (dotnet/runtime#82662) change in runtime (which was added in .NET8 Preview3), WindowStyle = Hidden flag has a different behavior. After this change, the order of events received by child process windows (such as Activated, Loaded, etc.) was different which caused few assertions to fail.

If we set WindowStyle to normal (default) or remove the hidden flag, the test starts behaving correctly.

@jeffschwMSFT - We are inclined to change our tests to align to this new behavior. Would like to know if this approach is correct?

A reduced sample repo is attached below :

Run_WindowTest.zip

(You can comment the WindowStyle = Hidden flag to restore the old behavior)

[jeffschwMSFT]

Great investigation. Adding @jeffhandley to see if the change in behavior is expected, or if this is an issue/unintended breaking change.

[rchauhan18]

@jeffhandley - awaiting your inputs

[jeffhandley]

I filed dotnet/runtime#89910 for this and marked it as a regression from .NET 7. I think it's likely we'll revert the change made in #82662, but I'll give the crew involved in that change an opportunity to look for a fix.

[jozkee]

We are inclined to change our tests to align to this new behavior. Would like to know if this approach is correct?

@rchauhan18 removing the WindowStyle.Hidden flag will give you back the previous behavior before dotnet/runtime#82662 so I think that's the right thing to do. I don't think there was any real reason for you having UseShellExecute = False and WindowStyle = Hidden since that didn't achieve anything prior to the change.

[miloush]

If we changed a documented lifecycle of a Window, then this needs to be taken a bit more seriously. If, however, the test code assumes that window must be activated in order to render content, as it was suggested originally, then the test should be fixed.

[batzen]

IMHO, the test used something that never worked.
Now that it started working the wrongly written test fails.
The test would also have failed and received wrong results, at least I guess so, when it had used UseShellExecute = True and WindowStyle = Hidden.
Has anyone from the WPF team tried that combination?
If it also fails that way, just fix the test.
If it passes that way, the WPF team should investigate why that's the case.