TL;DR; Thanks to Clojurists Together, Calva now has a new, powerful, and much more maintainable, REPL window.
(The rest of this post is sort of a mix of a project report and some self-therapy, with me trying to sort things out by thinking out loud.)
The Project
When Clojurists Together announced that they would this summer accept applications for smaller bursts of project maintenance, a k a Summer of Bugs, I got reminded about the biggest source of problems in Calva: its REPL window. If we could get rid of that window, we would get rid of a lot of bugs, quirks and unmet user expectations.
I therefore applied, suggesting I could take a shot at creating a new REPL window for Calva, one built on top of the regular Calva editor, in which so much work has been put to create a nice Clojure experience.
In my application I made it clear that it would be mostly an experiment, to see if the idea was within my reach to realize. It could end in me failing to find a path towards it and the funding would then only have provided some learnings. CT accepted my application on these conditions, something I think speaks volumes about what a wonderfully humane organization it is.
The Problem with the REPL Window
When I released Calva v2, a big part of that was this new REPL window. It is a very ambitious thing, a whole separate web browser based editor hosted in a VS Code webview. You can think of it as a Clojure focused Code Mirror. With this we could create any experience we wanted and we used it to create a pretty nice Clojure REPL editor. It was a joy releasing and announcing it, and also to receive mostly super positive feedback and actionable suggestions for improvement.
Cutting a rather long story short: this REPL window turned from being a nice project to a never ending string of hard to fix quirks. A whole new editor is a huge undertaking. An example I can give is when VIM Extension users expect the REPL editor to be VIM-ish, and aware of their configurations. It could easily eat up half a year of dedicated work to come even close. (I never tried, half a year is just a guess, haha.)
What’s more:
- Most work with the REPL window has been put into making the its editor behave as similar to the VS Code/Calva editor as possible.
- I, myself, don’t find all that much value in a REPL separate from the source files I am editing. Rich comments are my cup of tea.
- I, myself, is not the kind of developer that can maintain such an ambitious project as the REPL window is. It needs focus and special skills. As it is now it spreads me super thin instead.
All that said, the way the REPL window editor code is factored, a lot of the work being put into it has benefitted the regular Calva editor. (We wouldn’t have created the new Calva Paredit otherwise, for instance.) In hindsight it is pretty clear that Calva wouldn’t be as nice as it is today, without this whole-new-editor initiative. So, no regrets here, just learnings. Learnings and an insight that working this hard on making the separate editor work identically to the regular one – mostly failing at it – is suggesting something. 
Results
Full disclosure: We don’t have those paired source <-> REPL files yet. We have, however, positioned things so that this should be pretty easy to implement. I think that with some feedback on what we have just added, the whole paired thing might turn out better than I had first envisioned. We’ll see. Let’s look at what we do have.
In trying to create a plan for this Calva Summer of Bugs project, I went to my main source of input and ideas for how Calva should work, my co-maintainer, Brandon Ringe. After hearing me out and pondering the idea some he suggested that I start with the way Calva outputs evaluation results. Using the raw text output buffer made it a bit less helpful than it could be. Using a regular Calva editor for it could make a lot of difference, and shouldn’t be all that much of work.
Said and done. I started in that end and quickly had it happening. When 90% was done, of course there were 90% more to do to make it smooth and feel as part of a nice workflow. Part of this was to solve an issue with printing side effect output as it arrived. Lucky for me, Brandon had some time to spend on it and he sorted it out for me.
So, then I had a minimum piece of improvements to Calva to show for our efforts.(Quite a lot of improvement, at that.) But I could still add some more.
As regular Calva editors are REPL connected, this output window was automatically sort of a REPL. Asking the friendly people in the #calva Slack channel to test it out, one of the first things suggested was that if it had a notion of current namespace, it would be pretty useful as a REPL window.
So I added the current namespace notion to the window, which brought on the need for displaying the namespace. For that I decided to use the familiar prompt concept. It took me a while of fiddling, but two days later I had something that looked like a complete replacement of the current REPL window. Apart from a bit less rich stacktrace display, it is way better than the current REPL window, as far as I can tell.
Because it is mainly a regular Calva editor, some of the things the new output window sports are:
- The same Clojure syntax highlightning as the regular files
- Rainbow brackets and indent guides
- Paredit
- REPL connection, making it an alternative to the REPL window for explorations (which was the hoped for goal of the Summer of Bugs effort). It is treated as a CLJC file, so the user can toggle between the Clojure and ClojureScript REPL at will.
- A Clojure debugger (made possible by previous CT funding)
- VIM extension support – a huge win over the current REPL window.
Demo of the debugger active in the new output window:
It also has rich Stacktraces. It was not worth printing stack traces to the previous output channel. Much thanks to the CIDER underpinnings, the traces has clickable links to the file locations in the stack frames. Lacking the old REPL window’s HTML capabilities, this one is printed as a Clojure/EDN vector, making it easy to navigate using Paredit, or filter, using Clojure.
Demo of an output window stacktrace with clickable file locations and peek hovers:
Lessons Learned
The rationale behind making use of the previous work on the Calva editors to make this output/repl window extra powerful proved sound.
However, VS Code was not made with LISPs in mind and it sometimes feels like a battle making it support Clojure coding. Most of the things I thought would be easy turned out quite tricky, and I had to yield to the VS Code API and adapt several of my intended features a bit more than I liked.
The VS Code API sort of has two modes:
- Either you build things that are envisioned by the VS Code team, and get full support from the API, resulting in a very consistent look-and-feel. I think this is part of why VS Code is so nice and snappy.
- Or you are pointed towards the Webview API and can create about which experience you want, but it can mean a lot of work. (This is the old REPL window approach.)
Building something that should work a lot like the regular editor with the Webview approach is a lot of hard and frustrating work, just to meet the floor of the expectations that the users bring to an editor.
This new REPL window instead takes the first approach and bends it. Hard. Sometimes it doesn’t allow bending without breaking, so there are some usage quirks left, but all-in-all I’d call a project success. This new window offers almost everything that the old one does, and then some. Once I have mustered some more courage, the old window will be completely removed.
I also learnt that even though the Calva code base is a bit messy, it is still very supportive of Clojure IDE development. It is a bit of a mixed bag then, things that I thought would be easy, turned out hard and things that I worried would be hard, showed to be easy. I guess that is just regular software development.
Please Take it for a Spin
I hadn’t expected to be able to release this new window in an non-experimental way within the scope of the project. But it turned out I got payback for a lot of the work I and others have put into Calva before and things developed more quicker and more smoother than I had hoped.
If you update the Calva extension in your VS Code install, you will be using the new output/REPL window once you have connected Calva to your project. Please provide feedback and suggestions for further improvement.
See https://calva.io/output/ for the first version of the docs for the feature. I’d be happy to receive feedback on those as well.
