Had some interesting discussions with various people about the performance of clojure libraries in clojutre last week. Now that we have the excellent cljdoc, could we also solve the measuring of performance of clojure libraries too? Some dedicated servers to run the tests, collect the data and plot in cljdoc?
We have been perf (regression) testing and tuning our own libs, but currently on developer laptops. I guess many other languages/ecosystems have something for this already?
I know nothing about the whole infrastructure necessary, but I love the idea. It could foster a culture of contributing benchmarks to prove some things about libraries speed, and not “blindly” believing the authors words. Funded by Clojure Together?
I don’t have anything to contribute to the infrastructure question, but want to mention an alternative to Criterium, jmh-clojure.
What’s interesting about it is that you can declare your benchmarks as data in a jmh.edn file, and run them using lein jmh. Automatic perf testing infrastructure could simply look for a jmh.edn file in the project root, and would know how to run the benchmarks and collect the result data. And with it being based on JMH you get the best possible confidence in the accuracy of the results.
(Anecdote: I first used Criterium for performance testing of a library that I wrote. When I switched to lein-jmh, the Criterium results were more or less confirmed, so anyway Criterium seems to be accurate enough.)