After a long break from speaking at conferences and user groups, I’ve decided to give a talk about REPL-Driven Development at a couple of (virtual) user group meetups over the next couple of months (since this is a topic I’m always pushing here on ClojureVerse and on Slack etc).
Most of the “talk” is going to be a live coding session, building a small web application directly via the editor/REPL, with minimal tooling to show how I go about it. I want both presentations to be interactive so, although I have an outline for the live section, I want to encourage folks to ask questions and suggest things they’d like to see covered during the talk.
I figured I’d solicit some input here as well, in advance: what would you like to get out of a talk about RDD with (mainly) live coding to show various techniques?
Hi Sean, really interested in the talks, are these meetups I have to pay for or is it some exclusive club thing?
I’ve been following and using your Chlorine config and spoken with you a few times on the Clojurian Slack under my real name Rowan Barnard.
Always interested in learning more about your development process in Clojure.
As for feedback, I’m not sure I’m your target audience myself, having no industry experience but I would like to know how a live development process affects the methodology of writing software.
Was just interested in that as I was reading a while ago about continuous integration in some article where they claimed many people saying they were doing this were really doing what they dubbed “continuous build” at best. And that really the whole way you write software has to change to enable continuous integration within an organisation.
So it had me thinking that Chlorine REPL development seems to enable a very live dev experience that should be conducive to continuous integration, so what kind of development process would you need to enable “true” continuous integration and how would an organisation’s development methodology need to change to enable this?
Perhaps some of this is outside the scope of your talk though.
And thanks again for making all your Chlorine config stuff available on Github
Continuous Integration and Deployment aren’t really related to the REPL.
Continuous Deployment is the idea that as soon as you merge code into the shared master git repo, it should be deployed to production.
Continuous Integration is the idea that you should always be working with the most up to date version of the code base, not just the version that was there when you started making your changes. Thus CI is about you pushing your partial changes into the shared git repo and pulling down the shared master git repo’s latest change on a daily basis. So everyone working on the code base is always in-sync or close to it.
The combination of the two would be that you push to the shared master on a daily basis code that will be able to go to prod without breakage, and you pull from it on a daily basis.
This means you get to keep on top of all the integration and deployment, and so you don’t get yourself into a deployment or integration hell.
The REPL is a way for you to run your code as you write it, calling into it in arbitrary order and into any part of it even if it’s not designed to receive user input that way.
You can obviously combine all three approaches if you want.
CI/CD stuff is definitely out of scope for this talk but I sort of get what they mean: nearly all CI systems are not actually continuous, they are discrete test/build cycles that are triggered frequently (and sometimes in parallel, and sometimes so frequently that they end up running “almost continuously”).
Clojure’s REPL certainly allows for an application to run continuously and accept updates in real-time, but for a lot of changes they would need to be applied in a very specific order (bottom-up) and interleaving tests to validate those changes is a complex process to automate – and our current feature-based development process, with commits that are often “groups” of related changes, is not granular enough for a truly continuous integration process to work.
We run Socket REPLs in several of our processes in both our pre-production and production environments at work, so we do have the ability to apply those granular changes – manually – and we do sometimes do that when we want to apply a bug fix without actually deploying a new build (even though our deployment is automated for most of our processes, we have a couple of “singleton” legacy processes that require a little downtime due to a more complex deployment process). Mostly we use those REPLs for investigation and debugging.
Regarding Chlorine, I’ve recently switched from Atom to VS Code because Chlorine’s author has ported it to VS Code, as Clover, and I’ve posted my VS Code/Clover config to GitHub as well as my Atom/Chlorine config. The functionality is identical and so are the key bindings I use, so it’s been an easy transition, but the surrounding functionality of VS Code and its ecosystem is much greater than Atom.
The ideas behind CI and CD are only tied to commits on a shared repository because that’s the implementation we base things on and the granularity of daily pulls and pushes is definitely up for debate and, I would argue, not core to CI/CD.
If your team commits changes 100 times a day, CI should run the test/build process 100 times a day, and CD should mean 100 production deployments a day.
If we could connect our editor changes and REPL evaluations to that pipeline instead, with a sufficiently smart test-on-demand system, based on what each change meant needed to be tested, we could indeed have all those continuous changes flow directly into a truly Continuous Integration process and then into a truly Continuous Deployment process. We can dream. And maybe Clojure, with it’s unit of compilation being the function (essentially), could fulfil that dream at some point?
Looking forward to it!
A live coding session is exactly the right thing to show. There will probably be plenty of subtle differences and improvements over the way work.
Now, I’m not sure the extreme versions of the core ideas of CD and CI are actually the best ones in practice, but part of that could simply be that we need to find a way to make it practical.
As I recall, those ideas came out of XP, which obviously was all for eXtremes. XP really loves making everything continuous, I actually wish they’d have called pair programming continuous code review, feel that would have had a stronger impact on people if they named it that.
Within XP, you might be able to achieve finer granularity, if you truly do TDD for every single function and pair program all things, use feature flags, I can see how you could maybe get to a place where you integrate and deploy every changes that can compile. Still, I feel this isn’t practical quite yet.
If we could connect our editor changes and REPL evaluations to that pipeline instead, with a sufficiently smart test-on-demand system, based on what each change meant needed to be tested, we could indeed have all those continuous changes flow directly into a truly Continuous Integration process and then into a truly Continuous Deployment process
And this is super inspiring. Because I never really thought, why should Clojure leverage the practical tooling that other languages had to setup to enable them practically apply CD and CI practices? Can’t Clojure do something better here?
It made me think:
What if we had a shared REPL? What if the whole team connected to the same REPL? What would it take to achieve this and make it practical? Could we design a REPL that enabled this and made it practical?
What if feature flags could be built into Vars and our REPL ? When I send a form to the REPL, what if instead of swapping the Var root, it like added itself to the Var and a feature flag.would control which root you’d be getting ? What if it even had nice A/B testing functionalities built in like running both and comparing their result, or running B X% of the time and A the other. Etc.
You can certainly have multiple people connect via REPLs to the same running image. I often have more than one separate REPL session into the same remote process (via Socket REPLs).
But I suspect the rest of that “dream machinery” would need program analyzing features that are currently beyond us or, at best, are still in heavy research (for example, if tests fail at some point, how would the process know how/when to start applying changes again and how would we deal with a period of time during application of those changes where the system is in a “failing” state?). Automatic feature flagging is an interesting idea but would have to account for multiple changes to the same var, possibly by multiple developers, and an entire set of feature flags might need to be activated atomically, in order to “switch on” a consistent set of changes, and those automatic feature flags would need to be GC’d at some point once it is determined that they’re either superseded or no longer possible to roll back. That’s some fascinating stuff. But we should probably not derail this thread with that discussion…
Ah thank you very much for the edification Sean and Didibus
Well I will look forward to the presentations regardless, I am sure they will be interesting and I will learn new things.
It made me think:
What if we had a shared REPL? What if the whole team connected to the same REPL? What would it take to achieve this and make it practical? Could we design a REPL that enabled this and made it practical?
What if feature flags could be built into Vars and our REPL ? When I send a form to the REPL, what if instead of swapping the Var root, it like added itself to the Var and a feature flag.would control which root you’d be getting ? What if it even had nice A/B testing functionalities built in like running both and comparing their result, or running B X% of the time and A the other. Etc.
I suspect those things sound nicer on paper that they are in practice. If we take the premise that simplicity is at the heart of how we do things in Clojure, I’d say these are non-starters. Having a “google docs” experience on a shared REPL sounds like a pain (what if you need to refresh your namespace? do you wipe everyone else’s stuff as well?), and you’d also need to keep track of those flags to know which var you’re actually referring to? Sounds way too complex…
The main thing I’d like to see, which I haven’t seen from other REPL videos, is how you minimize keystrokes. For me, RDD requires a lot of jumping around between files/windows/expressions to make sure everything gets reevaluated. I’ve found the feedback loop for TDD to be faster because it doesn’t require that. Even the 2+ second startup time for Clojure is faster than all the jumping around. I’m assuming there’s something I’m missing or I’m not using the tools right.
Well, that’s part of why I wanted to do this talk/demo and show folks how I avoid a lot of that “jumping around” and how to (mostly) ensure that you don’t need to re-evaluate a whole chain of stuff when you are editing a single, low-level function… I typically have my REPL running for days (sometimes even weeks) so startup time doesn’t even factor into my workflow.
Ah yeah for me this will be super useful also, I probably spend too much time manually doing a lot of this editing stuff and jumping around as richard_heller puts it. I’m hardly expert at either Atom or VS Code so I’m sure there’s lots of ways I can do things faster.
On Slack, David Reno asked me to show how I use tap> for debugging, so I’m adding this note here as a reminder for myself – and to jog people’s memories here in case there are other things they’d like to see.
I’d like to see how REBL and Reveal fit into your workflow and where they can come in handy, if that’s not too outside the scope of the talk. I’ve been using REBL with your Chlorine deps file but other than as a handy way of viewing my map layouts don’t really know how useful it can be. So would like to see where it comes in most useful and maybe where Reveal shines the most in comparison to REBL.
I switched from REBL to Reveal once I had a way to have it automatically display values as tables (as REBL does by default). I’m planning to tweak that a bit and integrate it with my deps.edn setup (which will probably make an interesting blog post!).
REBL isn’t open source or programmable so Reveal wins in that area. My talk/demo will definitely show some of the things I do with Reveal that perhaps aren’t obvious
(for example, if tests fail at some point, how would the process know how/when to start applying changes again and how would we deal with a period of time during application of those changes where the system is in a “failing” state?). Automatic feature flagging is an interesting idea but would have to account for multiple changes to the same var, possibly by multiple developers, and an entire set of feature flags might need to be activated atomically, in order to “switch on” a consistent set of changes, and those automatic feature flags would need to be GC’d at some point once it is determined that they’re either superseded or no longer possible to roll back. That’s some fascinating stuff. But we should probably not derail this thread with that discussion…