This feels like a rather basic project setup issue. I just wanted to learn from the gurus and get some ideas for the best way to develop libraries in tandem
I write a lot of little programs and scripts in Clojure and those naturally have lots of little helper functions. As you write your main script/application you end up tweaking the helpers, covering new corner cases you expose etc. But these helper functions end up being copied around haphazardly between scripts and apps. Things like functions that convert formats or make plots etc. Obviously this isn’t ideal!
I’d like to split these independent helper functions off into their own mini libraries
One workflow that I’ve seen is the “monorepo” setup. This is the design used by thing/geom. However I think long term this has proven to be a bit problematic b/c Karsten Schmidt ended up having a whole ton of awesome features… but they’re all wrapped up and hidden away in the mono repo. He has a whole matrix library there, and renderers and things for working with 3D geometry etc. etc. But I’m guessing there is a discoverability issue and maybe people don’t wanna have to pull in the whole kitchen sink to access one tool in some corner
Instead I’m trying to find a nice deps.edn way of doing this but I don’t really get what the language designers’ expected workflow is.
But how do you do both at once…? I obviously need to tweak and add features to the libraries as I work on the application Should I have a local development deps.edn that points at local folders and then another deps.edn with remote repos to distribute? (If so, what’s the best way to set that up with Cider?)
Or can I have one deps.edn that looks locally and then falls back to remote repos if it doesn’t find anything?
Or maybe there is a better alternative I haven’t considered?
As an alternative, take a look at https://github.com/ingydotnet/git-subrepo. Given how it’s a common problem that’s language-agnostic, IMO using git-subrepo along with tools.deps would be the best approach. Of course, you still have to refer the subrepos as local dependencies to take their own dependencies into account.
I add source paths in a leinengen project file sometimes, so you can hack on sibling projects simultaneously. This is useful, however it can lead to issues with keeping the adjacent code consistent with what is actually released/deployed. It does solve the convenience issue. I have not migrated to deps.edn, although the source-paths approach could work there as well (possibly under a different alias).
We develop 18 applications simultaneously in a monorepo. We’ve adopted Polylith as the structure for that but there are other approaches that work well for different people. See my series of blog posts on https://corfield.org about monorepos for how our usage has evolved over time.
You don’t need multiple deps.edn files: just different aliases. One alias can bring in the dependencies as :local/root for development and another alias can bring in the dependencies via Clojars/Maven for building (or the latter can be the default and you override them via a :dev alias to get the local versions).
Yeah, you’ve got a good point @joinr . the crux of the issue is that there isn’t any straightforward way to ensure local and remote versions of your mini-libs don’t get out of sync. So you’ll end up overlooking a git push and what works on your machine is actually broken for people that pull it. I don’t really see how this can be worked around neatly
I think the monorepo setup prolly makes sense if you’re working within an org
I’ve sorta thought it over again and I think realistically I will probably mostly want to share utility libraries. No one needs my crappy one-off scripts and whatnot (and if they do, they can patch the dependencies). So I’ll work local-first and then if it’s something worth sharing I’ll sync everything up and write a proper distributable deps.edn
Thanks for the insight guys. and helping me think through it. I appreciate it
I will second @seancorfield on using Polylith. We use it effectively for this type of thing. It’s an effective way to use a monorepo structure, but support many different targets. It’s also great for code re-use across projects.
It takes work, though, to get things into the shape that Polylith wants. It might be more opinionated than what you’re looking for @geokon-gh.
Should I have a local development