Would like a version of `clojure.test/is` which aborts the `deftest` if fails

[Jim_Newton]

This post raises the subject discussed here on clojurians

Slack is your productivity platform

The issue is that would like to have a version of is which causes the current test to abort, but allows the next test (next test defined by deftest) to continue.

Here is a real example from one of my projects.

(deftest t-plus-associative
   ;; check associativity
   (doseq [p1 polynomials
           p2 polynomials
           p3 polynomials]
     (is (= (sut/+ (sut/+ p1 p2) p3)
            (sut/+ p1 (sut/+ p2 p3)))
         (format "Discovered non-associative input for poly-plus\np1=%s\np2=%sp3=%s"
                 p1 p2 p3))))

The problem with this test is that if the sut/poly-plus function itself raises an exception, then each is simply fails, registers the failure, and the next iteration of the doseq loop continues. I would rather that such an error would cause t-plus-associative to abort, and the next test (defined by another deftest to run.

I have fixed this in my project by writing my own version of is which simply re-throws the exception after registering the failure. I’ve inserted the expression (throw t#)

(defmacro util-try-expr
  "modified copy of clojure.test/try-expr"
  [msg form]
  `(try ~(assert-expr msg form)
        (catch Throwable t#
          (do-report {:type :error, :message ~msg,
                      :expected '~form, :actual t#})
          ;; Now, re-throw the same exception, so the testing framework
          ;; will abort the current test, considering it a failure.
          (throw t#))))

(defmacro is
  "Drop-in replacement of clojure.test/il which aborts the current
  test if the expression is found to be false."
  ([form] `(is ~form nil))
  ([form msg] `(util-try-expr ~msg ~form)))

I think it would be useful to have an alternate to is which allows a test to fail and finish, rather than explicitly forcing the failing test to continue.

[Jim_Newton]

Summarizing some of the comments in the clojurians thread: Someone suggest that I should just modify the test, and debug my code then re-install the test. However, this is not really a good approach. In my case, I am now trying to debug my own code. Rather I am am trying to write a test capable of testing case which someone else wrote. The test should give reasonable feedback in the presence of a buggy implementation of the functions in question.

When the test is written using clojure.test/is, a particular user’s code triggered 75 million lines printed to the log file. Since the test in question was run in batch in a docker image using using lein test configured to use lein-test-report-junit-xml, the docker image never actually finshed generating the xml file before the system ran out of memory (or maybe timed out, not sure which). And the user never received any feedback at all of what was failing.

[Jim_Newton]

Another suggestion is to rewrite the doseq to use some sort of (is ...(every? ...)). That’s a good idea in principle, but it is not clear how to generate the error message because the 2nd argument to is will not be in the scope of the p1, p2, p3, of the three concentric calls to every?