Improving error messages in Clojure (as a library)?

Some example output for the ArityException using Maria’s library:

Here’s a type error:

51

A great deal is possible by catching exceptions and applying templates to them. Some of the other things you mention are more complicated because they require interacting with more of the context of the error (for instance knowing what symbols are in scope in order to offer “did you mean?” suggestions), but nothing here is impossible, especially if all the tooling providers worked together.

3 Likes

I’m very sympathetic to the idea of improving error message. I’m beginning to be sensitive to the idea that what error messages appear should be context dependent, that there isn’t a one-size-fits-all answer that might be implied by a “single library approach”. The errors I see in the repl could very well be different from those that appear in logs, which are different which would appear in tests, for example. And depending on what I’m working on in the repl, I might want either more terse or more expansive, or more more interactive error messages.

What I think would be a really good step forward is to have a best practices on supplying custom error messages in a way that’s composable with other “error message authors”. I’d like to be able to use something expound, some error message libraries supplied by Maria, and others supplied by @borkdude while also allowing me to add my own.

I’m not sure what that looks like, but I’d like to see something that doesn’t require buy-in to the exclusion of other tools.

I’m afraid a plethora of tools won’t help the beginning user that is not yet familiar with the ecosystem. A lot of interested newcomers will have moved on long before they have assembled a helpful dev environment which can be quite time consuming because of the error messages they will encounter… which defeats the purpose.

5 Likes

Oh, very much agreed. It would be good to have a good solid recommendation to get someone started. That said, we shouldn’t limit ourselves to only the new user experience. Users should be able to mix the tools they want to use, adding as they become more experienced, for example, or dropping those that served as training wheels while they were getting started.

That’s currently true in spec, but expound has a workaround.

(def email-regex #"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,63}$")

(expound/def :example/email (s/and string? #(re-matches email-regex %)) "should be a valid email address")

(expound/expound :example/email "sally@")

;; -- Spec failed --------------------
;;
;;   "sally@"
;;
;; should be a valid email address

Of course, this requires that the authors write error messages (or consumers write error messages after the fact). I understand this would be more widely adopted if it was in the spec library, but nonetheless, it’s something that anyone can add today.

1 Like

Would it be possible to extract the (expound) messages from other sources such as from the Spec :problems map? Maybe a configuration option into Expound? We are having a custom :reason field injected into problems for this. Should the spec errors be templates or just strings? "number should be in between 1 and 6" is kind of nicer better than "number is not in the range".

1 Like

That’s a good idea! The second arg to expound/def could either be a string or a function that is passed the problem (and maybe the spec itself?) and returns an error string. I believe that would allow you to grab any information you’ve added to the problem and would also allow you to include any relevant information (e.g. the range information) in the message.

I’ll create a GitHub issue to explore this.

Hey everyone!

I’m so glad you’re interested in improving error messages. I for one and entrirely optimistic that it can be done well. I’ve even started an issue to discuss it here https://github.com/always-be-clojuring/issues/issues/7 at always-be-clojuring. I just think it will take quite a bit of brute force. Lots of writing and testing.

I won’t go deeper here because I covered the topic in the github issue I linked to above. If you’re interested please take a look at the next steps.

Rock on!
Eric

3 Likes

I would replace the default error printer with a new one. It will take some experimentation and lots of hard work, but there are some possibilities for dispatching to the appropriate function. There’s the exception type and you could also use a regex on the message.

Core.specs will help with type errors in arguments. But there’s a long way to go to make error messages that give hints for how to fix the problem. There are some errors that are going to take some cleverness to swap out.

That said, I believe it’s just a matter of elbow grease.

Rock on!
Eric

1 Like

These are some very challenging errors you’ve found. I don’t think we’ll find a general solution that will cover all errors. It will have to be a patchwork of approaches. I don’t think any of these are difficult when attacked directly.

One more tool related to this: Bruce’s awesome Strictly Specking. Great for data-driven apps, as it also suggests how to fix things. Some discussion on it’s future here.

Would like to see a “standard” error keys for spec to emerge: many ways to close key-specs, one tooling to print the errors & suggest fixes.

1 Like

I don’t mean to come across as all bitter about it or anything, but with the release of the most recent Clojure survey I went back through the previous results, and it was galling to see error messages come across as the most frustrating thing about Clojure for three years in a row, and mentioned in every survey stretching all the way back to 2011 (though you occasionally need to dip into the raw data to find it). IMO the addition of specs for the core libraries in 1.9 has made error message output easier for machines to parse but harder for human beings to read, so we don’t seem to be going in the right direction here.

1 Like

Agreed, core specs can only help with type errors in args and there are many other types of errors that need to be explained clearly.

In the case of spec errors, Expound will support examples soon(ish):

(require '[clojure.core.specs.alpha])
(require '[clojure.spec.alpha :as s])
(require '[clojure.spec.test.alpha :as stest])
(require '[expound.alpha :as expound])
(set! s/*explain-out* (expound/custom-printer {:print-specs? false})) ; The "relevant specs" are very long, so let's omit them

(s/fdef foo :args (s/cat :x string?))
(defn foo [x]
  x)
  
(stest/instrument)

(foo :hello)
;;clojure.lang.ExceptionInfo: Call to #'user/foo did not conform to spec:
;;                            form-init8427266847984625578.clj:1
;;
;;                            -- Spec failed --------------------
;;
;;                            Function arguments
;;
;;                              (:hello)
;;                               ^^^^^^
;;
;;                            should satisfy
;;
;;                              string?
;;
;;
;;
;;                            -- Example ------------------------
;;                            (<f> "hello")
;;                            -------------------------
;;                            Detected 1 error

The important part is (<f> "hello"). Note that that the example includes <f> instead of foo because Spec doesn’t currently include the function name in the explain-data

If anyone is looking for a low-investment way to contribute to error message handling, one avenue is to find unhandled ClojureScript errors produced in maria.cloud and report them in our error handling wiki. Just break some code and add a heading and example of it to the wiki.

We work on the errors in bursts so it might be a while to get a particular message looked at. The errors will be ClojureScript-specific, but many specifics and most patterns are shared across Clojure dialects.

3 Likes

To be fair, making errors easier for computers to read is the right first step on the way to making them easier for people to read. If they have a predictable format we have a much better chance of writing code to present them well, plus we get other unrelated benefits around logging and other automated processing.

5 Likes

Would it make sense if Spec helped a little more than it does right now to produce human readable output?
E.g. when a predicate returns non-true and it’s not a simple type check, it would be nice to know what the semantics of that predicate exactly is.

1 Like

I’m not sure if I’m the only one - I love Spec, but when I see one of those error messages, my head and my eyes hurts. :sob:

1 Like

The issues with Clojure error messages are numerous and as I’ve said before, I think it’s important to try to tease them apart. Some of the things I have thought about for a long time:

  • Message context
    • Invalid values often propagate through the outer clojure.core layer into the JVM layer before they blow up. That code is often either very generic or lacking outer context. My experiments with clojure.core function specs significantly improve this - the error is caught at the first function layer and an error can be throw in the user’s call with location information, the bad predicate, the value, etc.
    • Macros obscuring context - given one or more layers of macro expansion an error sometimes occurs in expanded code, which exists neither in the user’s code nor in the macros
    • Missing language context - I see a class of complaints that are really about not having a good mental model for how some part of Clojure works. That’s not the user’s fault. Probably a relatively small set of these could be made gentler and make a big difference (like trying to invoke things that aren’t functions). In any case, having a means to integrate additional information would be useful. I don’t imagine that information would be “in core” but we could provide the hooks.
  • Conveyance
    • There is only one exception type that conveys error location (file/line/col) right now: Compiler$CompilerException. Forcing all location wrapping through this single inner class is confusing and at odds with the better support we now have in things like ex-data. Due to this, CompilerException is often used to wrap better errors, merely to provide location information.
    • There needs to be a distinction made between “your invocation is invalid Clojure code”, “your invocation produced an exception in your code”, “Clojure compiler error”, “Clojure runtime error”, etc. Maybe not exactly those, but a handful of broad categories. At the point of throw or rethrow, the general category is known and Clojure should help more here (so that tools can help more on the receiving end).
    • Code has a context (local bindings etc) and we know a lot of it. Conveying more of that could be put to great use.
  • Reporting
    • Due to the CompilerException wrapping above, there is often a chain being sent purely for location conveyance. Tools generally don’t unwrap these exceptions, but they should. Cursive for example shows you only the top error, the CompilerException, which has the location information, but usually a less useful error message and a useless stack trace. It is almost always more useful to look at the root cause exception - the message and stack are almost always what you actually want (but missing the location info!)
    • Unnecessary stack traces - some (maybe most) of the categories above don’t need actually need a stack trace at all. If you had the right info for a language error or spec error, you can just tell the user the thing they did and why it’s wrong (and maybe how to fix it).
    • Unnecessary frames - when stack traces are emitted, there is often a lot of noise and many people have taken stabs at simplifying. I find I hate most of the “simplified” stack trace printers but I have hope that there are built-in improvements to be made.
  • Spec
    • We are well aware that the core spec macro errors are intimidating and in some cases worse than prior behavior. (However, they also catch and fail on many previously accepted invalid inputs.) The spec macros like ns and defn have a) wide fanout (like the ns clauses), b) complicated structure (destructuring) and/or c) recursion (destructuring). In many ways, these represent the hardest possible cases for automatic error reporting. I think there are many many ways these messages can be greatly improved for these hard cases. I think people miss that the “easy” cases are really pretty good a lot of the time (since we don’t have the simpler core function specs included yet).
    • expound has done a great job building on the explain-data to go really far in making spec messages super friendly. Our goals in the core language differ somewhat from expound and while I do expect the default printer to get much better, I don’t expect core to end up where expound is now. In particular, there seems to be a lot of pressure for customizing errors and I actually think there is way more room to make the non-custom cases really good, and there is a ton of benefit in that.
    • I think there are still vast opportunities for tools to leverage specs and explain-data visually in novel ways beyond what expound can do. I think we can do more in spec in the future to provide some help here.

I’ve got some other writeups on this stuff and I probably forgot some things but that’s a good start.

34 Likes

@alexmiller Thanks for the excellent, detailed analysis.

My assumption is that specs will be the foundation for better error messages. But we haven’t been able to really confirm this since most of core remains un-specced. It may be true that spec errors (or libraries that build upon spec like Expound) still fail to generate good errors in common cases and more changes or features are necessary.

These specs are especially important for beginners since they won’t know how to spec their own functions initially. A larger set of specs would make a REPL like https://gist.github.com/bhb/2686b023d074ac052dbc21f12f324f18 much more useful.

There have been a few 3rd-party attempts to build a set of specs for core, but as you’ve noted, these are incomplete and buggy. Do you have any sense of when a broader set of specs may be released? From my perspective, I’d learn much more about error messages from an update to core.specs.alpha than from any particular spec bug fix (or even a major reimplementation of spec).

Thanks for all your hard work on spec and elsewhere!

3 Likes