[Idea] A tool for helping developers get started using Clojure CLI tools and deps.edn

lilactown · May 17, 2019, 10:31pm

Hey all. I’ve been marinating in an idea for a short while now and wanted to get some feedback on it. The TL;DR is: an easy CLI tool that would wrap common deps.edn aliases, with an easy and simple escape hatch for when you’re ready to move on from it.

The problem I’d like to try and solve is: I think there’s a hole in the tools-deps side of things, where leiningen and other tools work super well, like packaging projects, generating projects, adding dependencies, etc. The nice thing about having them in one umbrella tool is that all of those commands are easily discoverable.

Currently there are many people developing excellent applications that fit those use cases and work as aliases. I’m interested in the discoverability and ease, esp. for new developers, who I work with a lot at work. I’m more talking about the developer UX.

My proposed solution would be to have a utility that could provide easy access to a collection of “default” commands for people getting started with tools-deps. It would essentially serve as a wrapper around tools-deps aliases, with some functionality to update and then “inject” them for customization in particular projects.

Here’s an example. I’ve named the utility plum for the extent of this message

$ plum update # pulls the latest utility aliases and installs them to your global deps.edn
Pulling latest...
Backing up /Users/will/.clojure/deps.edn to /Users/will/.clojure/deps.edn.bak.
Installing aliases into /Users/will/.clojure/deps.edn.
Success!

$ plum new app foo.core # alias for seancorfield/clj-new

$ cd foo

$ plum test # alias for com.cognitect/test-runner
Running tests in #{"test" "src/test/clojure"}

Testing foo.bar-test

FAIL in (a-test) (bar_test.clj:7)
FIXME, I fail.
expected: (= 0 1)
  actual: (not (= 0 1))

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.

The novel part of this is the inject command which will take one of the aliases, and insert it into the local deps.edn file to allow for customization

$ plum inject test
Adding new alias :plum/test. This will modify your deps.edn. OK? (Y/n)
y
Adding alias.
Success!

You can then modify the local alias and run the modified version with plum test, or via clojure -A:plum/test. it does not require e.g. team mates to have the tool installed to use these same commands.

I think that this would act as a nice spring-board for users who are newer to the tools-deps ecosystem - perhaps they’re coming from Leiningen, or other languages with similar kitchen-sink tooling - while giving them an easy way to eject (well, inject ) once they’ve outgrown the simple commands and want to customize their projects more.

I’ve created plum here which is a representation of my current ideas around it. Feedback would be appreciated. Does this sound like a good idea? Would people be willing to contribute?

My hope, if this is something that would add value, would be that this would live in clj-commons or something where the community would be able to participate in what aliases we include, and ensure that proper attribution is given.

orestis · May 18, 2019, 8:45am

I had a similar idea before, and discussed it briefly around Slack. I think (obviously) that it makes a lot of sense, and I will certainly take a closer look.

I was also looking for some non-clojure stuff integration, e.g. using something like https://github.com/casey/just

mdiin · May 18, 2019, 11:43am

I think this is a very good idea, which fills a hole in the ecosystem. From previous experience, it was easier for my coworkers to pick up on leiningen than tools.deps, simply because it has more of a kitchen sink approach.

teodorlu · May 18, 2019, 11:45am

I like the idea. Two quick comments:

If this is a tool for learning, what is the final aim? People maintaining their own ~/.clojure/deps.edn? Or should they keep using this? Related: emacs dotfile management; using a community driven distribution like Spacemacs vs maintaining one’s own. I’d like to know what your endgame is.
I don’t want /usr/local/bin/plum linking to ~/.plum/plum. I prefer using ~/.local/bin/plum instead – to keep proper user separation. In the end of the plum update function, there’s a line hard-coding plum to that location, which gives an error on my system (that would require sudo on Ubuntu).

lilactown · May 18, 2019, 11:29pm

My end game is: provide an easy solution for project-level tasks that works for a lot of people, up to a point, and then give them the resources to customize their project to its needs, without having to support every single use case directly.

My aim is provide something of a “standard library” for doing operations on a Clojure project. Common tasks like project generation, adding dependencies, testing, linting, formatting, building, publishing should have an “easy” route that covers the basics. Once someone needs more than the tool offers, it should be easy and simple to move on towards something more bespoke.

Put another way, many build-tools for specific languages quickly approach reinventing make. I think that once you need to fine tune a project’s tasks and really customize things, using a general purpose tool like make (or just, or gradle, or bazel, or…) is a better option.

My thoughts are also inspired by tools like create-react-app in the web dev world, which allow you to quickly scaffold an application. The specific tools and configuration is setup for you and largely opaque. Then, when you eventually out grow the features and options it provides out of the box, you can run an eject command that brings all of those tools and configuration to the foreground and allows you to customize based on your project’s growing needs.

Emacs dotfile distributions like spacemacs is kind of similar, but my goal is that plum kind of lives alongside these other build tools rather than replaces them. It should compose well with make or other bespoke (even just a set of shell scripts) systems that might be calling clojure directly. It should be easy to hand a project that someone built using plum to someone who doesn’t run plum.

RE: the installation machinery. Please open an issue! The current method is pretty brutish and should definitely allow people to install based on their platform’s best practices.

didibus · May 20, 2019, 4:55pm

Where are you pulling the aliases from?

I had thought that building a package manager out of tools.deps would be nice. Something where you could do:

plum install foo/bar

And that just adds the alias to your global tools.deps file, as well as add a launcher for it on your path. So one could then do:

bar x y z

Assuming foo/bar is a scripting util.

And you could update them all:

plum update

Or specific ones etc.

This is a bit of a different goal it seems. Since you’re looking more to build a standard build tool out of tools.deps. Where I was thinking more as a standard way to package Clojure CLI apps for people to install and use.

But it also feels to me like both goals could be combined. Where plum could be a build tool, and an app installer/updater/runner. Which all under the hood just uses tools.deps.

My only thing was I didn’t know how to handle the aliases global repos. For example, if I’m the author of a tool, and want to publish it, what would I do, so that plum would now know about it inside its repository?

I was thinking a standard config file. Like say a plum.edn file at the top of a git repo. Where the app developer configures the alias in it, and some meta for creating the proper launcher for it, and a version number, etc. And then some repository, where the first one would just be github.

So when you run:

plum install foo/bar

It would look for a repo foo/bar on github, and from it, it would look for a plum.edn file. The downside is searching and listing wouldn’t be possible, but for a start it seems pretty nice.

That way, you end up with something more akin to npm or pip in UX, but all built on the tools.deps machinery.

Anyways, bottom line, I like your idea, and I do think there’s potential here for a good thing.

P.S.: It would be nice if plum itself was installed as an alias, instead of having to clone the repo.

lilactown · May 20, 2019, 5:17pm

Where are you pulling the aliases from?

The aliases are currently enumerated in the plum.update namespace: plum/clj/src/plum/update.clj at master · lilactown/plum · GitHub

My hope is that I can get some feedback and create some governance around what commands are available and what aliases they depend on.

I had thought that building a package manager out of tools.deps would be nice.

I think that’s a really interesting idea, but as you said doesn’t really solve the same problem I’m targeting: making an easy on-ramp for users who are not already familiar with the entire Clojure ecosystem (or who don’t care to learn as much) a set of tools to do “regular” operations like run tests, install dependencies, generate new projects, and build jars.

Tools like npm and crate serve both uses. Certainly plum could be extended to serve both of those uses, but it would not currently be high on my list of priorities. If it’s something you’re passionate about, maybe we can take it over DM or another thread and figure out a plan for how that might fit into the grand scheme of things.

It would be nice if plum itself was installed as an alias, instead of having to clone the repo.

That doesn’t seem to be in line with the point of the tool? Everything that the plum does under the hood, other than pulling the repo for the latest version and displaying the help file, is done via aliases. The plum executable is simple a shell script that wraps clojure -A:plum/<command-you-entered>.

If you don’t want the executable, you could run the plum.update main yourself, using git deps, and then run the global :plum/* aliases via clojure yourself.

lilactown · May 20, 2019, 5:28pm

If I think about what plum should be, it would contain at least the following commands:

Update to latest version of commands: plum update (Exists)
Add an alias to a project-local deps.edn: plum inject (Exists)
Add new dependencies: plum add clj-time "0.15.0" (Exists)
Generate new projects: plum new template my-org/project-name (Exists)
Start a REPL in a project: plum repl (Exists)
Run tests: plum test (Exists)
Format code according to a standard: plum format (Does not exist)
Build a jar / uberjar: plum build (Does not exist)
Publish a library to a maven repo: plum publish (Does not exist)

Some of these commands exist, but might not be in an ideal state. Some of them don’t exist at all.

My call to action to interested readers:

Is there anything missing from the above list of features?
Help me find good solutions for them. Some of them might already exist, some of them might not.

If you’re interested in helping out, I have created a Clojurian Slack channel #plum for live discussion. You can also create issues and pull-requests on the repo.

didibus · May 21, 2019, 5:04pm

This is just an idea. But if the shell script was greatly simplified so the install instructions could just be something like:

printf "#!/bin/sh\nclojure -Sdeps {...}/\"$@\"" > ~/.plum/plum

ln -s ~/.plum/plum /usr/local/bin/plum

And then plum update would update that file with the latest sha.

That way, you don’t need git, just the Clojure CLI, and plum is itself bootstrapped with tools.deps.

Dunno, maybe it’s not better, just kinda brainstorming.

Also, wouldn’t you want a way to remove a dependency? And maybe a way to update the version of an existing dependency, as well as being able to add a git or local dependency.

lilactown · May 21, 2019, 5:35pm

git is currently the easiest way I could think of to have a mechanism for users to update as well as allowing rollback / checkout of alpha releases or other branches. We may find there are reasons not to use that, but for the moment seems fine?

I agree that removing dependencies might be nice as well (though not something I reach for super often). on installing things other than mvn versions. Updating to a latest version might be useful, but trickier. All of these simply need to be implemented

didibus · May 21, 2019, 6:01pm

Hehe, ya sorry, I’m doing a lot of thinking out loud, and not really helping with any of the work Plum just got me excited and dreaming about possibilities.

lilactown · May 21, 2019, 9:08pm

Pull requests welcome!!!

pmonks · May 21, 2019, 10:00pm

This list looks great. Any thoughts about plugins for plum?

I realise plugins are a slippery slope, but if done properly they do have the benefit of allowing some development to happen semi-independently of the core plum code base.

lilactown · May 21, 2019, 11:41pm

What would plug-ins do, exactly?

pmonks · May 22, 2019, 1:53am

What would plug-ins do, exactly?

Anything that isn’t in the command list above (and arguably that list ultimately as well). If commands were pluggable, it would allow the community to extend plum in ways you may not have envisaged or have time / interest to develop yourself, without having to work on the core codebase to make them available to users of the tool.

tl;dr - the main benefit is to decentralise and democratise development of new commands.

lilactown · May 22, 2019, 4:43pm

Well, atm plum executes clojure -A:plum/$@ so you can certainly add your own “plum commands” to any deps.edn:

 :aliases {:plum/foo ...}

And execute via plum foo.

I’m not sure how much ceremony I want to add to that right now, since I want to ensure that we can support the canonical set of commands at least.

I think it would be easier and simpler to direct users that want to provide similar commands that might extend beyond plum’s defaults to use clojure directly, e.g.:

 :aliases {:foo ...}

Executed by clojure -A:foo.

martinklepsch · May 22, 2019, 8:01pm

This is a super interesting proposal! I think it addresses a bunch of pain points with the current CLI tools, covering problems that have previously only been adequately solved by Boot and Leiningen.

This is a maybe getting ahead of things a bit but one thing I’d think might be interesting to consider in the context of upgrade-ability is something that Boot does really well. For every Boot task there is a symmetry between CLI invocations and Clojure forms:

boot push --pom pom.xml --file some.jar --gpg-sign

(push :pom "pom.xml"
      :file "some.jar"
      :gpg-sign true)

This example is slightly contrived but this upgradeability (into real code) is super nice when you want to eventually combine multiple steps into one. As soon as you outgrow the “run individual commands in sequence” approach I think this is a great affordance. This becomes particularly apparent when you want to combine multiple long-running commands at the same time, like running tests and cljs compilation on filesystem changes.

I think kaocha implements a similar system but @plexus will be more knowledgeable about that.

I don’t think this should be anywhere near the top in terms of priorities but maybe it is something that could be considered for generally being in scope of plum.

Keep it up, excited to see this become a thing! The reference to create-react-app (from the little I know about it) really makes sense and having some comparable tooling in the Clojure ecosystem would be awesome for anyone getting started with Clojure!

markx · May 23, 2019, 1:53am

I would really love to have this! I was hoping for something like pip, npm, go get, or cargo, which you could use to install an executable project (it pulls the dependencies and puts an executable in your PATH).

I’m kinda new to the JVM ecosystem, and surprised to find out that to use an app built in Clojure, I have to somehow get the uberjar (build from src or download with maven), and manually create a shellscript for it and put to PATH.

arichiardi · May 23, 2019, 2:50am

Hi all!

I am reading and reading this and can’t help but thinking that while it would for sure solve some issue with the current tools.deps setup, it would be awesome if this idea would percolate back into clojure itself

In any case good job there!

And a way to share common aliases to “trusted” libs (like clj-new), nice! I like it! And my local deps.edn will definitely become smaller.

arichiardi · May 23, 2019, 2:56am

Just found out a similar, yet different, project on GitHub: https://github.com/matthias-margush/aka