Cancelling tests that take too long

Projects Kaocha
1

Does Kaocha have a way for cancelling tests that run too long? Especially with asynchronous code you can have tests that never finish. For example, I just had a test like this:

(deftest promise-test
  (let [my-promise (promise)]
    (is (= 1 @my-promise))))

This test will block indefinitely because my-promise is never delivered. It’d be handy if Kaocha could after some configurable time limit mark the test as failed and continue to the next test (or fail fast).

2

That’s not currently there, although I agree it would be useful. kaocha-cljs does have this, as there the execution model is by its nature asynchronous. We wait for certain events coming off a queue to transition through a state machine, and if the next state transition takes too long we time out, although I believe in that case we skip remaining tests, as it assumes that at that point your ClojureScript environment has become unresponsive so there’s no use in trying to run more tests.

I think the hard part is dealing with cleaning up. You would want to interrupt the thread of the test that you assume is blocking, and there are some caveats when doing that on the JVM. That said I think it’s 100% doable, it could even be a plugin, using a pre-test hook to add a :kaocha.testable/wrap function which runs the inner testing logic on a separate thread, and sets a timer which cancels the thread and fails the test.

2 Likes
3

Hi. I’m running into this same issue as well. I’m running a very large test suite, and one of the tests is freezing due to async code never finishing. Because the test suite is large and randomized, it is difficult to tell which test is actually freezing using the default reporter. When the tests freeze, I have to cancel it with Ctrl-C, and I only get a report of tests that might have failed during the incomplete run, but since the frozen test did not technically fail, I get no information from it.

Since it’s not currently possible to have a test timeout, Is there any way that I can at the very least tell the reporter to print out what tests, or even test namespaces, are being run before each test is run (or just after the tests have been cancelled with Ctrl-C) so that I can better narrow down which test is the problem? Even something that reported which test was being run at the time of cancellation would be a big help.

Thanks for an awesome library!

1 Like