Cancelling tests that take too long

Does Kaocha have a way for cancelling tests that run too long? Especially with asynchronous code you can have tests that never finish. For example, I just had a test like this:

(deftest promise-test
  (let [my-promise (promise)]
    (is (= 1 @my-promise))))

This test will block indefinitely because my-promise is never delivered. It’d be handy if Kaocha could after some configurable time limit mark the test as failed and continue to the next test (or fail fast).

That’s not currently there, although I agree it would be useful. kaocha-cljs does have this, as there the execution model is by its nature asynchronous. We wait for certain events coming off a queue to transition through a state machine, and if the next state transition takes too long we time out, although I believe in that case we skip remaining tests, as it assumes that at that point your ClojureScript environment has become unresponsive so there’s no use in trying to run more tests.

I think the hard part is dealing with cleaning up. You would want to interrupt the thread of the test that you assume is blocking, and there are some caveats when doing that on the JVM. That said I think it’s 100% doable, it could even be a plugin, using a pre-test hook to add a :kaocha.testable/wrap function which runs the inner testing logic on a separate thread, and sets a timer which cancels the thread and fails the test.

2 Likes