Support for validating the content of recorded Issues

[dabrahams]

Description

Users shouldn't have to concoct their own versions of this facility. You're going to need it for your own tests anyway; it should be exposed.

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

swift-testing version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

[stmontgomery]

Thanks @dabrahams. swift-testing currently has a set of APIs named withKnownIssue which are meant to serve this purpose. See documentation at:

https://swiftpackageindex.com/apple/swift-testing/0.1.0/documentation/testing/expectthrows#recording-known-issues-in-tests

Can you check whether withKnownIssue meets your needs? If not, I'd love to know what functionality it's missing.

[dabrahams]

I think while those might do what I am asking for (I am not sure), their names certainly imply that these issues are supposed to get fixed at some point. The facility I'm talking about is expressly for testing that some testing facility works, and should reflect the intent that the proper operation of that facility under the circumstances is to report a test failure. The intention expressed by a piece of test code is important, and I would say withKnownIssue expresses the wrong thing to a reader.

[stmontgomery]

Any chance you could provide a small example of the usage scenario you have in mind? Just to help us understand your use case a bit better. Are you attempting to test a library which provides testing utilities of its own?

[dabrahams]

@stmontgomery Did you look at the README on the page I linked to? I think it answers all those questions and indeed describes some behavior you might want for withKnownIssue as well. Is there some information missing from the README that I can add?

You can see more robust examples of where this function is useful in these tests of the StandardLibraryProtocolChecks package.

[grynspan]

Do you have a specific interface in mind here, @dabrahams?

[dabrahams]

Not really; I haven't internalized your interface style.

[stmontgomery]

Thank you. I think we understand the use case better here, now. We think it could be useful to improve this area, but consider this fairly niche and not a priority for the project while it's still in an early stage of development. We will hold this Issue for further consideration, and in the mean time, if you would like to propose a specific SPI or API enhancement along these lines, we can consider it further. I'll also re-title this issue to try to clarify how it's different than a general-purpose "expected failure" API, which for most users is withKnownIssue.

For now, we do not plan to take action on this ourselves, but I do think withKnownIssue may suit your needs in the short term, regardless of its name.

[grynspan]

If we extend what we've done with #expect(throws:), it might be natural to have:

#expect {
  ...
} records: { issue in
  ...
}

As an even more general form of:

#expect {
  ...
} throws: { error in
  ...
}

I do worry that we're creating too many similar-but-subtly-different interfaces, but maybe I worry too much?

[SeanROlszewski]

Presumably, the second block is to codify their expectations against the issue?

#expect {
  thisThrowsAnErrorAndIWantThat()
} records: { issue in
  #expect(issue.value == "whatever")
}

[grynspan]

The second closure would return true if the issue matched what was expected to be recorded, and false for other issues. This is how #expect {} throws: {} handles errors. So you'd have the opportunity to validate the issue was what you intended to record.

Alternatively, we could maybe introduce something like:

withKnownIssue(intentional: true) { ... }

Which would behave like withKnownIssue() today, but would not record an XFAIL state for intentional issues.

[SeanROlszewski]

I think the variant with the second closure is both easier to understand and more flexible, since someone could author an arbitrary boolean expression to evaluate the issue, which is more useful than always saying, "This should produce an issue." We can also automatically fail the test if it didn't produce an issue.