Include generated input with each test.chuck checking block error

[zjroth]

CIDER includes generated input in reports when the generated input is made available, which happens when using test.chuck and its checking block. Currently, generated input is made available only to the first test condition within a checking block. Unfortunately, this means that the generated input is not reported anywhere in CIDER unless the first test condition fails. As a result, the generated input is (in my experience) typically not reported in CIDER.

This PR addresses that issue by modifying the cider.nrepl.middleware.test/current-report atom selectively based on whether a test.chuck testing context is available, which results in the presence of a non-nil value in the :gen-input key of the report precisely when the report corresponds to an error generated in a checking block.

cc: @daemianmack

[alexander-yakushev]

I don't think I ever used test.chuck (just the regular test.check). Could you please make a small reproducer to make it possible to validate the changes? A cider-nrepl test case would be nice too, but we can start with a reproducer and see if a proper test can be written.

[bbatsov]

@zjroth ping :-)

[zjroth]

Thanks for the response, @alexander-yakushev, and for the ping, @bbatsov. Apologies for the delay in this unfortunately busy season for me.

Could you please make a small reproducer to make it possible to validate the changes? A cider-nrepl test case would be nice too, but we can start with a reproducer and see if a proper test can be written.

I'm not familiar with the term "reproducer" as used here, but it seems to be something like a minimal example of code for which the issue arises. Is that correct?

[zjroth]

As an example of how this situation appears to me, consider the following test:

;; [clojure.test.check.generators :as gen]
;; [com.gfredericks.test.chuck.clojure-test :as chuck :refer [checking]]
(deftest show-input-in-test-chuck-checking-block
  (checking "Generated value shown in CIDER test report (only first test fails)"
    {:seed 1}
    [x (gen/choose 1 10)]

    (is (even? (* x x)))
    (is true))

  (checking "Generated value not shown in CIDER test report (first test succeeds but subsequent test fails)"
    {:seed 3}
    [x (gen/choose 1 10)]

    (is true)
    (is (or (= 1 x)
            (even? x)))))

For me, this produces the following CIDER test report results:

Fail in show-input-in-test-chuck-checking-block
Generated value shown in CIDER test report (only first test fails)

expected: (even? (* x x))
  actual: (not (even? 9))          
   input: [{x 3}]

Fail in show-input-in-test-chuck-checking-block
Generated value not shown in CIDER test report (first test succeeds but subsequent test fails)

expected: (or (= 1 x) (even? x))          
  actual: false          

Note the missing input from the second failure above.

With this PR, the test results change to the following:

Fail in show-input-in-test-chuck-checking-block
Generated value shown in CIDER test report (only first test fails)

expected: (even? x)
  actual: (not (even? 3))          
   input: [{x 3}]

Fail in show-input-in-test-chuck-checking-block
Generated value not shown in CIDER test report (first test succeeds but subsequent test fails)

expected: (or (= 1 x) (even? x))          
  actual: false          
   input: [{x 7}]

Note that each failure now contains an input value with the dynamically generated input, which is not always discernible from the actual or expected value.

[zjroth]

Hi, @bbatsov and @alexander-yakushev. I'm just checking back in on this. Is my example above sufficient as a reproducer as requested? (NOTE: I modified the example above slightly to make it clearer—in a somewhat contrived way—that the expected and actual values do not always convey the dynamically generated input.)

[alexander-yakushev]

Hi! Sorry for such a long delay. I've tried it out and can confirm that it works. Thank you!