Improving error messages in Clojure (as a library)?

clojure
#6

…which does leave open the possibility for non-traditional funding mechanisms, like academia, grants from government or business, and community efforts like Clojurists Together.

#7

I think a library that combines maria/friendly and expound (and configures both) could go a long way here, but there are a few challenges.

First, I think it’s key that the presented error messages are fairly consistent to avoid a jarring jump in presentation when a user goes from a non-spec error message to a spec error message. This either requires coordination between maria/friendly and expound or configuration options for one or both.

Second, such a library would greatly benefit from fairly wide adoption from intermediate and expert Clojurists to give feedback on missing and unhelpful errors. While beginners have an invaluable perspective on what is confusing and helpful, I don’t think it’s realistic to ask them to give feedback online when they are just starting out.

And finally, spec errors are only useful if functions are specced. We would need library owners to adopt spec (specifically, using fspec for public functions) and more clojure.core specs in https://github.com/clojure/spec.alpha.

#8

I’m not sure a library will be able to solve this issue. Some of the bad error messages are attributable in part to the inherent design of Clojure.

  • Compile errors might need the compiler to be smarter at specifying what an issue is, and suggesting alternatives. Most other compilers would catch typos, and suggest to you the correctly typed version instead like: defun does not exist, did you mean defn ? or Unmatched delimiter ), did you mean (let [a 12] (+ a 10))) ?
  • Specs don’t take a human explanation of errors, so it just says: you failed this predicate, here's the predicate. Contrast this with most other assertions libraries, and they have you give it a reason, so that it could say: you failed this predicate, because Value of :name can only contain Latin characters, for predicate ...
  • Most errors are actually thrown from Java. For example, if you type ([1 2 3]) you get ArityException Wrong number of args (0) passed to: PersistentVector, but most beginners don’t realize Clojure has types, it should say: ArityException Wrong number of args (0) passed to function: [1 2 3] at line 55 for ([1 2 3]). Or when you see IllegalArgumentException Key must be integer clojure.lang.APersistentVector.invoke Clojure isn’t even trying here, this is an exception thrown by the Clojure Java implementation. Its really confusing, because why is there a call to something called invoke? A beginner is really confused, again, it should point to the user code: IllegalArgumentException First argument of function [1 2 3] "Key" must be an integer. At line 55 for ([1 2 3] [2]) You have a lot of these, where you see the error from inside the Clojure implementation, instead of at the user level. This I believe is the most fundamental issue with Clojure error messages. The errors should be about Clojure, not the underlying Java details.
  • Lots of things are not errors. (defn add [a b] "A function that adds a with b." (+ a b)) Please, at least give a warn for this dammit. I don’t want guard rails, but can I have headlamps at least?

I think these require changes to the Clojure implementation itself, and maybe to Spec. Libraries seem to only pretty-print errors, or highlight parts of it that are more relevant, but that’s a bandaid. Maybe I’m wrong, and a library could fix some of those, but I’m not sure how.

#9

Some example output for the ArityException using Maria’s library:

44
Screen Shot 2018-03-15 at 18.59.44.png1034x224 12.2 KB

Here’s a type error:

51

A great deal is possible by catching exceptions and applying templates to them. Some of the other things you mention are more complicated because they require interacting with more of the context of the error (for instance knowing what symbols are in scope in order to offer “did you mean?” suggestions), but nothing here is impossible, especially if all the tooling providers worked together.

#10

I’m very sympathetic to the idea of improving error message. I’m beginning to be sensitive to the idea that what error messages appear should be context dependent, that there isn’t a one-size-fits-all answer that might be implied by a “single library approach”. The errors I see in the repl could very well be different from those that appear in logs, which are different which would appear in tests, for example. And depending on what I’m working on in the repl, I might want either more terse or more expansive, or more more interactive error messages.

What I think would be a really good step forward is to have a best practices on supplying custom error messages in a way that’s composable with other “error message authors”. I’d like to be able to use something expound, some error message libraries supplied by Maria, and others supplied by @borkdude while also allowing me to add my own.

I’m not sure what that looks like, but I’d like to see something that doesn’t require buy-in to the exclusion of other tools.

#11

I’m afraid a plethora of tools won’t help the beginning user that is not yet familiar with the ecosystem. A lot of interested newcomers will have moved on long before they have assembled a helpful dev environment which can be quite time consuming because of the error messages they will encounter… which defeats the purpose.

#12

Oh, very much agreed. It would be good to have a good solid recommendation to get someone started. That said, we shouldn’t limit ourselves to only the new user experience. Users should be able to mix the tools they want to use, adding as they become more experienced, for example, or dropping those that served as training wheels while they were getting started.

#13

That’s currently true in spec, but expound has a workaround.

(def email-regex #"^[a-zA-Z0-9._%+-][email protected][a-zA-Z0-9.-]+\.[a-zA-Z]{2,63}$")

(expound/def :example/email (s/and string? #(re-matches email-regex %)) "should be a valid email address")

(expound/expound :example/email "[email protected]")

;; -- Spec failed --------------------
;;
;;   "[email protected]"
;;
;; should be a valid email address

Of course, this requires that the authors write error messages (or consumers write error messages after the fact). I understand this would be more widely adopted if it was in the spec library, but nonetheless, it’s something that anyone can add today.

#14

Would it be possible to extract the (expound) messages from other sources such as from the Spec :problems map? Maybe a configuration option into Expound? We are having a custom :reason field injected into problems for this. Should the spec errors be templates or just strings? "number should be in between 1 and 6" is kind of nicer better than "number is not in the range".

#15

That’s a good idea! The second arg to expound/def could either be a string or a function that is passed the problem (and maybe the spec itself?) and returns an error string. I believe that would allow you to grab any information you’ve added to the problem and would also allow you to include any relevant information (e.g. the range information) in the message.

I’ll create a GitHub issue to explore this.

#16
#17

Hey everyone!

I’m so glad you’re interested in improving error messages. I for one and entrirely optimistic that it can be done well. I’ve even started an issue to discuss it here https://github.com/always-be-clojuring/issues/issues/7 at always-be-clojuring. I just think it will take quite a bit of brute force. Lots of writing and testing.

I won’t go deeper here because I covered the topic in the github issue I linked to above. If you’re interested please take a look at the next steps.

Rock on!
Eric

#18

I would replace the default error printer with a new one. It will take some experimentation and lots of hard work, but there are some possibilities for dispatching to the appropriate function. There’s the exception type and you could also use a regex on the message.

Core.specs will help with type errors in arguments. But there’s a long way to go to make error messages that give hints for how to fix the problem. There are some errors that are going to take some cleverness to swap out.

That said, I believe it’s just a matter of elbow grease.

Rock on!
Eric

#19

These are some very challenging errors you’ve found. I don’t think we’ll find a general solution that will cover all errors. It will have to be a patchwork of approaches. I don’t think any of these are difficult when attacked directly.

#20

One more tool related to this: Bruce’s awesome Strictly Specking. Great for data-driven apps, as it also suggests how to fix things. Some discussion on it’s future here.

Would like to see a “standard” error keys for spec to emerge: many ways to close key-specs, one tooling to print the errors & suggest fixes.

#21

I don’t mean to come across as all bitter about it or anything, but with the release of the most recent Clojure survey I went back through the previous results, and it was galling to see error messages come across as the most frustrating thing about Clojure for three years in a row, and mentioned in every survey stretching all the way back to 2011 (though you occasionally need to dip into the raw data to find it). IMO the addition of specs for the core libraries in 1.9 has made error message output easier for machines to parse but harder for human beings to read, so we don’t seem to be going in the right direction here.

#22

Agreed, core specs can only help with type errors in args and there are many other types of errors that need to be explained clearly.

In the case of spec errors, Expound will support examples soon(ish):

(require '[clojure.core.specs.alpha])
(require '[clojure.spec.alpha :as s])
(require '[clojure.spec.test.alpha :as stest])
(require '[expound.alpha :as expound])
(set! s/*explain-out* (expound/custom-printer {:print-specs? false})) ; The "relevant specs" are very long, so let's omit them

(s/fdef foo :args (s/cat :x string?))
(defn foo [x]
  x)
  
(stest/instrument)

(foo :hello)
;;clojure.lang.ExceptionInfo: Call to #'user/foo did not conform to spec:
;;                            form-init8427266847984625578.clj:1
;;
;;                            -- Spec failed --------------------
;;
;;                            Function arguments
;;
;;                              (:hello)
;;                               ^^^^^^
;;
;;                            should satisfy
;;
;;                              string?
;;
;;
;;
;;                            -- Example ------------------------
;;                            (<f> "hello")
;;                            -------------------------
;;                            Detected 1 error

The important part is (<f> "hello"). Note that that the example includes <f> instead of foo because Spec doesn’t currently include the function name in the explain-data

#23

If anyone is looking for a low-investment way to contribute to error message handling, one avenue is to find unhandled ClojureScript errors produced in maria.cloud and report them in our error handling wiki. Just break some code and add a heading and example of it to the wiki.

We work on the errors in bursts so it might be a while to get a particular message looked at. The errors will be ClojureScript-specific, but many specifics and most patterns are shared across Clojure dialects.

#24

To be fair, making errors easier for computers to read is the right first step on the way to making them easier for people to read. If they have a predictable format we have a much better chance of writing code to present them well, plus we get other unrelated benefits around logging and other automated processing.

#25

Would it make sense if Spec helped a little more than it does right now to produce human readable output?
E.g. when a predicate returns non-true and it’s not a simple type check, it would be nice to know what the semantics of that predicate exactly is.