Test failures that arise when inference starts succeeding #50620
Description
Quite a few packages seem to have tests that looks like this:
@test_broken @inferred foo(args...)The intent here, I think (and I'm sure I've written some of those myself), is to say "I'd like inference to to succeed for this task, but it doesn't currently, so I'm marking this as something to keep track of." The @test_broken is acting like a try/catch on the error (one that prints a summary of the number of such failures at the end of the test run), preventing the failure in inference to cause the aggregate test suite to fail.
But now consider what happens when inference improves, as indeed it has in Julia 1.10, and the @inferred no longer errors. Now the @test_broken receives the return value of foo(args...), which means it will pass only if it returns false. A characteristic error is the one like in https://github.com/JuliaArrays/AxisArrays.jl/actions/runs/5456747193/job/15230694492?pr=222,
Core: Error During Test at /home/runner/work/AxisArrays.jl/AxisArrays.jl/test/core.jl:275
Expression evaluated to non-Boolean
Expression: #= /home/runner/work/AxisArrays.jl/AxisArrays.jl/test/core.jl:275 =# @inferred op(C; dims = 1)
Value: [6 15 … 33 42]
For anyone trying to figure out what needs to be done to make PkgEval happy, this pattern is probably worth remembering.
To be clear, these are package errors: the test should be written @test_broken @inferred(foo(args...)) == outputvalue. When inference starts succeeding, this too will fail, but it will fail "in the expected way" and a warning issued that the test should be changed to passing.
I'm posting this mostly to document the pattern and provide an explanation that can be cross referenced. I'm tempted to close this immediately after opening it, but I'll wait and see if anyone sees a real TODO here first.