Had an interesting conversation on Twitter about the state of error messages in Clojure
If the Clojure community would have a drop-in library which would turn “obscure” (for some value of obscure, this is subjective) stack traces/Spec traces into friendly error messages, without changing core, what would be the best strategy? So, I’m not talking about turning Spec output into human readable form only. Also the other errors that are not catched by Spec.
Would this help adoption of Clojure?
How do we get one “blessed” “better error messages” library that every beginner know how to find and use?
How do you perceive the state of error messages today? What are your “favorite” ones? What should they look like for beginners?
The error messages for Maria, which Stu mentions in that thread, are powered by the friendly lib, specifically the messages namespace, where we try to 1) wrap every error and 2) present the underlying problem in a friendly way. We’ve only barely started; there are lots more errors to find, and lots more field-testing to find out the best way to point folks to a solution.
IMO the path to a single library, if there is one, is for someone to put in the work and pull ahead of the pack in terms of error coverage, being pluggable to tooling, and replacement phrasing. (This puts me in agreement with Zach Tellman’s statements circa 1:33:00 of the podcast you reference in your twitter discussion with Stu.) Unfortunately it’s a lot of work and it’s not likely to be remunerated via normal means.
Ain’t nobody gonna get VC $$$ to make nice error messages. Ain’t no clients callin’ to hire you to make a Clojure stacktrace wrapper library.
Business and capital want ETL pipelines and ERP integrations and e-commerce websites and AI-driven news aggregator apps and business document storage and CMSes and everything else we make as production systems. Clear, effective error messages for both beginners and experienced devs are too far removed from production systems and business requirements. They’re more like Clojure being a lisp: it’s a great language feature, I think it’s important and valuable, but it’s not something that you can sell to non-developers. So ain’t nobody gonna pay for good error messages.
…which does leave open the possibility for non-traditional funding mechanisms, like academia, grants from government or business, and community efforts like Clojurists Together.
I think a library that combines maria/friendly and expound (and configures both) could go a long way here, but there are a few challenges.
First, I think it’s key that the presented error messages are fairly consistent to avoid a jarring jump in presentation when a user goes from a non-spec error message to a spec error message. This either requires coordination between maria/friendly and expound or configuration options for one or both.
Second, such a library would greatly benefit from fairly wide adoption from intermediate and expert Clojurists to give feedback on missing and unhelpful errors. While beginners have an invaluable perspective on what is confusing and helpful, I don’t think it’s realistic to ask them to give feedback online when they are just starting out.
And finally, spec errors are only useful if functions are specced. We would need library owners to adopt spec (specifically, using fspec for public functions) and more clojure.core specs in https://github.com/clojure/spec.alpha.
I’m not sure a library will be able to solve this issue. Some of the bad error messages are attributable in part to the inherent design of Clojure.
Compile errors might need the compiler to be smarter at specifying what an issue is, and suggesting alternatives. Most other compilers would catch typos, and suggest to you the correctly typed version instead like: defun does not exist, did you mean defn ? or Unmatched delimiter ), did you mean (let [a 12] (+ a 10))) ?
Specs don’t take a human explanation of errors, so it just says: you failed this predicate, here's the predicate. Contrast this with most other assertions libraries, and they have you give it a reason, so that it could say: you failed this predicate, because Value of :name can only contain Latin characters, for predicate ...
Most errors are actually thrown from Java. For example, if you type ([1 2 3]) you get ArityException Wrong number of args (0) passed to: PersistentVector, but most beginners don’t realize Clojure has types, it should say: ArityException Wrong number of args (0) passed to function: [1 2 3] at line 55 for ([1 2 3]). Or when you see IllegalArgumentException Key must be integer clojure.lang.APersistentVector.invoke Clojure isn’t even trying here, this is an exception thrown by the Clojure Java implementation. Its really confusing, because why is there a call to something called invoke? A beginner is really confused, again, it should point to the user code: IllegalArgumentException First argument of function [1 2 3] "Key" must be an integer. At line 55 for ([1 2 3] [2]) You have a lot of these, where you see the error from inside the Clojure implementation, instead of at the user level. This I believe is the most fundamental issue with Clojure error messages. The errors should be about Clojure, not the underlying Java details.
Lots of things are not errors. (defn add [a b] "A function that adds a with b." (+ a b)) Please, at least give a warn for this dammit. I don’t want guard rails, but can I have headlamps at least?
I think these require changes to the Clojure implementation itself, and maybe to Spec. Libraries seem to only pretty-print errors, or highlight parts of it that are more relevant, but that’s a bandaid. Maybe I’m wrong, and a library could fix some of those, but I’m not sure how.
A great deal is possible by catching exceptions and applying templates to them. Some of the other things you mention are more complicated because they require interacting with more of the context of the error (for instance knowing what symbols are in scope in order to offer “did you mean?” suggestions), but nothing here is impossible, especially if all the tooling providers worked together.
I’m very sympathetic to the idea of improving error message. I’m beginning to be sensitive to the idea that what error messages appear should be context dependent, that there isn’t a one-size-fits-all answer that might be implied by a “single library approach”. The errors I see in the repl could very well be different from those that appear in logs, which are different which would appear in tests, for example. And depending on what I’m working on in the repl, I might want either more terse or more expansive, or more more interactive error messages.
What I think would be a really good step forward is to have a best practices on supplying custom error messages in a way that’s composable with other “error message authors”. I’d like to be able to use something expound, some error message libraries supplied by Maria, and others supplied by @borkdude while also allowing me to add my own.
I’m not sure what that looks like, but I’d like to see something that doesn’t require buy-in to the exclusion of other tools.
I’m afraid a plethora of tools won’t help the beginning user that is not yet familiar with the ecosystem. A lot of interested newcomers will have moved on long before they have assembled a helpful dev environment which can be quite time consuming because of the error messages they will encounter… which defeats the purpose.
Oh, very much agreed. It would be good to have a good solid recommendation to get someone started. That said, we shouldn’t limit ourselves to only the new user experience. Users should be able to mix the tools they want to use, adding as they become more experienced, for example, or dropping those that served as training wheels while they were getting started.
That’s currently true in spec, but expound has a workaround.
(def email-regex #"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,63}$")
(expound/def :example/email (s/and string? #(re-matches email-regex %)) "should be a valid email address")
(expound/expound :example/email "sally@")
;; -- Spec failed --------------------
;;
;; "sally@"
;;
;; should be a valid email address
Of course, this requires that the authors write error messages (or consumers write error messages after the fact). I understand this would be more widely adopted if it was in the spec library, but nonetheless, it’s something that anyone can add today.
Would it be possible to extract the (expound) messages from other sources such as from the Spec :problems map? Maybe a configuration option into Expound? We are having a custom :reason field injected into problems for this. Should the spec errors be templates or just strings? "number should be in between 1 and 6" is kind of nicer better than "number is not in the range".
That’s a good idea! The second arg to expound/def could either be a string or a function that is passed the problem (and maybe the spec itself?) and returns an error string. I believe that would allow you to grab any information you’ve added to the problem and would also allow you to include any relevant information (e.g. the range information) in the message.
I’m so glad you’re interested in improving error messages. I for one and entrirely optimistic that it can be done well. I’ve even started an issue to discuss it here https://github.com/always-be-clojuring/issues/issues/7 at always-be-clojuring. I just think it will take quite a bit of brute force. Lots of writing and testing.
I won’t go deeper here because I covered the topic in the github issue I linked to above. If you’re interested please take a look at the next steps.
I would replace the default error printer with a new one. It will take some experimentation and lots of hard work, but there are some possibilities for dispatching to the appropriate function. There’s the exception type and you could also use a regex on the message.
Core.specs will help with type errors in arguments. But there’s a long way to go to make error messages that give hints for how to fix the problem. There are some errors that are going to take some cleverness to swap out.
That said, I believe it’s just a matter of elbow grease.
These are some very challenging errors you’ve found. I don’t think we’ll find a general solution that will cover all errors. It will have to be a patchwork of approaches. I don’t think any of these are difficult when attacked directly.
One more tool related to this: Bruce’s awesome Strictly Specking. Great for data-driven apps, as it also suggests how to fix things. Some discussion on it’s future here.
Would like to see a “standard” error keys for spec to emerge: many ways to close key-specs, one tooling to print the errors & suggest fixes.
