Announcing Kaocha, a new and improved Clojure test runner

[plexus]

For the last six months or so I’ve been working on Kaocha, a new and much improved Clojure test runner.

Why

There are a number of test tools available already, like lein-test, eftest, circleci.test, and some of them have pretty interesting features. The problem I see is that they are either bound to a specific build tool (lein, boot), or to a specific testing library (clojure.test, midje), or both. No matter which one you choose, there will be things you are missing. The current approach does not scale.

(is (= amount-of-work
      (* (count build-tools)
         (count testing-frameworks)
         (count desirable-features))))

Apart from that there are still features that are considered bog-standard in e.g. the Ruby world, that are not yet available anywhere in Clojure.

So I set out to create something that is feature-rich out of the box, and that is flexible and extensible. Support for other testing frameworks can be provided as an add-on (what kaocha calls a “test suite type”), and extra features can be provided as plugins.

What

Kaocha is both a tool and a base for tooling. It turns the imperative task of running tests into something that is data-driven, and takes care of the nitty gritty so that others can focus on the features they want to see.

So far Kaocha supports Leiningen and Clojure CLI, and can run clojure.test and (experimental/alpha) Midje tests. Features implemented so far:

• Filtering tests based on test names or metadata
• Watch mode: watch the file system for changes and re-run tests
• Randomize test order
• Detect when interrupted with ctrl-C and print report
• Fail fast mode: stop at first failure and print report
• Profiling (show slowest tests)
• Support for matcher-combinators
• Extensive documentation

How

After installing Kaocha you configure it create a tests.edn in the project root with your test setup. It describes the test suites you have, with their respective types (this determines how they are loaded and run). You can set various configuration flags in there, choose your output reporter style, and configure plugins.

After that you use Kaocha either from the command line or the REPL.

Your turn

Kaocha is being used in production by Nextjournal, and is used to run the Cljdoc tests, and of course I’ve been dogfooding it extensively. There is still a lot of work to be done to reach my original vision, but the foundations are looking really good. Now I need your help.

You can help by trying it out on your projects. If you have an open source project feel free to ping me and I’ll create a PR for you. Try it out and report back, both the positive and the negative. If you are missing something that prevents you from switching then open an issue, this will help me prioritize.

If you have a closed source project then you can hire me to set Kaocha up for you. Especially if you have multiple test suites (unit, integration, etc) I’d love to talk to you.

Your company can also fund me to make sure Kaocha reaches its potential. Get in touch and we can work something out!

[pesterhazy]

Very happy to see this. I’ve always thought that phpunit was much better than anything Clojure had to offer in terms of test runners, so it’s great to see progress in this area.

Is it possible (or desirable) to make this work with clojurescript as well?

[simongray]

Is the name from the Chinese 考察/考查 or is it just a coincidence?

[plexus]

Yes, that is indeed the origin of the name.

[plexus]

Great question! ClojureScript support is definitely on the roadmap, but it will be a big task with a lot of unknowns, and something that I’d hope to get funding for.

<speculative-fiction>

The way I imagine it would work is that Kaocha invokes the ClojureScript compiler, finds any test “vars” from the compiler env, then starts (or connects to an existing) JavaScript environment, and communicates with it over ClojureScript’s ReplEnv abstraction.

So Clojure-based Kaocha would still do the coordination, but the final step of running a test is delegated to the ClojureScript environment, which invokes the test in some kind of harnass to capture output, errors, return value, etc, and reports back.

This way ClojureScript tests become just another test type, and any plugins and tools can treat them equally

;; tests.edn
#kaocha/v1
{:tests [{:type :kaocha.type/clojure.test
          :id :backend-tests}
         {:type :kaocha.type/cljs.test
          :id :frontend-tests
          :kaocha.type.cljs.test/compiler-options {,,,}
          :kaocha.type.cljs.test/env :nodejs}]}

As you can imagine there will be many snags along the way, and there will be a significant gap between “working prototype” and “works robustly”, but I think this is doable.

</speculative-fiction>

I can imagine at some point there might be interest in a ClojureScript compatible version of Kaocha e.g. for use on bootstrapped environments. That is something I haven’t given much thought yet and that is not high on my list. Kaocha does a lot of JVM specific things regarding classpaths, dynamic vars, etc. so this would involve some pretty fundamental changes.

[ikitommi]

Looks great! As there are already many runners and utility libraries, should there be a comparison table of features? Maybe compare to good non-clojure runners too?

• https://github.com/bhauman/cljs-test-display
• https://github.com/weavejester/eftest
• https://github.com/metosin/bat-test

[plexus]

I’ll make sure to add a table as soon as I finish implementing whatever features the others have

More seriously I’d like to add a section to the docs that for each of them answers the question “I’m using $x, what’s in it for me when I switch?”

[orestis]

How is it that it’s the first time I’ve seen bat-test? We have to get some better google juice for a lot of things in the ecosystem…

[plexus]

I know! I only just discovered circleci.test, can’t believe it took me so long.

[timgilbert]

Just curious, how much of an effort would it be to integrate this with something like test2junit to get junit.xml output from it?

On a similar line, would it be possible to plug cloverage into it to get code-coverage reporting?

[plexus]

junit.xml is on the list and should be easy. In theory you can just add that library to the classpath and do bin/kaocha --reporter test2junit.junit/junit-report since it’s implemented as a standard clojure.test reporter. It seems it does need some extra state set up before so it might be a bit more involved than that, plus you face the general expression problem that you get with any custom clojure.test reporter when dealing with custom assertions, but all of that is solvable. circleci.test also has junit.xml output so I might poke my head in there for inspiration as well.

And again, if people really need this they can implement this as a plugin. The idea with Kaocha is that no single maintainer becomes a bottleneck for these kind of features, so go forth and extend.

Cloverage is high on the list, not least because I want to work on the coverage of Kaocha’s own test suite. Basic cloverage support is trivial, and was actually there before Kaocha underwent a big reorganization this summer. Cloverage doesn’t care much what runs the code once its instrumented, its --runner flag means you just need to extend a multimethod.

The limitation of this approach is that this means you’re running your tests through cloverage’s CLI tool, so you lose the ability to pass flags to kaocha. So I think the proper solution is a kaocha plugin which adds flags to kaocha to configure cloverage.

This also allows us to have cloverage configuration in your tests.edn, which is nice. This is also one of the reasons each test suite has a :source-paths option, so we know which code needs to be instrumented (the other reason is that --watch needs to know which files to watch.)

Longer term what I’d like to do with cloverage is allowing it to measure different levels of coverage. Currently cloverage only does “global” coverage, a line is considered covered no matter how it got called. One step up would be namespace coverage, where only tests that correspond with this namespace count towards coverage. The most stringent would be var coverage, where vars need to be fully covered by tests that target that var. (The final step would be mutation coverage, which I’ve had great success with in Ruby, but the tools aren’t there yet in Clojure, and frankly I don’t think the Clojure world is ready to aim that high. Baby steps.)

[plexus]

Some exciting Kaocha news, I will be funded from November through January to work on Kaocha thanks to Clojurists Together. I have a long list of things I want to work on. ClojureScript support is probably the biggest among them, so I think I will work on more general improvements the first month to get some momentum, then tackle ClojureScript support in December.

Meanwhile I just released 0.0-211 which already includes several improvements, including a deep-diff output. Compare the output of kaocha with the output of eftest.

Kaocha:

Eftest:

Hopefully these kind of tangible differences will encourage more people to try out Kaocha.

Somebody also asked if the xfail concept from PyTest could be brought to Kaocha. This is exactly the sort of things why plugins are there, so anything you’re missing can easily be added. I added an example plugin that implements xfail.

[plexus]

Junit.xml support is now available as a plugin: https://github.com/lambdaisland/kaocha-junit-xml

[sooheon]

Looks awesome, thanks for the contribution to the community.

I’d like to call attention to shadow-cljs as another build tool on cljs side that deserves attention.

[teodorlu]

After less than 5 minute from “I need colorized test output” to having a really nice terminal running bin/kaocha --watch, I really like the experience! Great work!