Lack of integration with system package managers is an obstacle for individual hobbyists

Community Center Watercooler
1

The entire clojure culture caters to proprietary enterprise software which is usually not released as open-source projects. Clojure is suitable for building uberjars and uberscripts that are deployed on mobile devices and web browsers and backend servers. Clojure is very uber.

But, to increase adoption, clojure needs to also cater to individual hobbyist developers who fiddle with various concepts on their local machines. If more hobbyists fiddle with clojure, they will be later available in the job market for clojure shops and clojure open-source projects. They will play with clojure by writing scripts for window managers or wayland compositors. Even on mac os and windows, hackers write system scripts for modifying the behaviors of their little systems.

In local computing environment, people generally use system scripting languages that allow them to install language libraries as user libraries or system libraries. System libraries are generally installed by system/OS/linux package managers. Perl and raku(perl’s cousin) have library paths for users, admins, and linux distros.

My local system scripts rely on language libraries installed by system package manager.

Babashka is okay for proprietary enterprise scripts that are later deployed as uberscripts because businesses typically don’t want to release them as open-source libraries later, but it is not good for “system” scripting because it doesn’t naturally integrate with system package managers as languages like python and janet do. If your babashka script relies on a third party library, then you will need bb.edn in the script directory or build an uberscript. That’s cumbersome for quick scripts that you use in your window manager.

Most people aren’t clojure shops and don’t work for clojure shops. Before they join clojure shops or create clojure shops, they need to play with clojure in their local environment against their own local systems. The existing clojure implementations add friction to that kind of fun play for individual hackers.

It seems that basilisp and jank can become system scripting languages if they later integrate with system package managers, but the maintainers don’t seem interested in system package managers. Basilisp maintainer is busy with his life. Jank maintainer is busy launching an alpha version of jank.

I like clojure as a language, but I, as a hobbyist developer, find myself using janet exclusively because janet was built from the scratch as a system scripting language. Everything is optimized for system scripting in janet.

Excellence as a language matters less than how suitable it is in an environment. My primary environment is my local linux system. Before people deal with bigger systems, they need to play in their personal environments.

2

Hi amano, welcome to clojureverse!

I’ve read your post twice now trying to understand what you are trying to achieve. Are you trying to understand how hobbyists use clojure for their desktop environments in the absence of system packages? Are you asking why no greater effort has been made to package clojure and clojure libraries in the way that you would like? Are you volunteering to do the work of packaging? (Which packaging system? Which projects?)

Or are you here to tell existing maintainers that they should redirect their time and resources into packaging things in a different way? If it’s that last one then this is not the place, or the way to go about that.

The clojure etiquette page states

They are not the place for advocacy about what ‘ought’ to be made. If you think something ought to be made, then make it. Otherwise, respect others peoples’ right to choose what they do with their time.

The reason that is there is that Open source is hard work, often for little reward. There’s always an infinite amount of things that could/should be done, built, fixed, documented, etc. every maintainer does a difficult exercise balancing priorities to decide what to work on on any given day. The last thing they need is someone barging in with little context, who doesn’t offer help, telling them their use case should get priority.

My suggestion would be something like this, package up some projects in the way you like to see them packaged. Write up your experience, find others who see the value, then reach out to projects to see if there’s interest from the core developers to help out, endorse, or whatever it is you need from them.

Show you have skin in the game, build a coalition, do outreach and advocacy. That way you’ll have a chance of making a difference. Simply posting the same strongly worded opinion piece to every single clojure forum on the internet, and going into discussion with anyone who doesn’t see the point, will not help your cause, to the contrary.

5 Likes
3

I actually tried several times to make linux distro packages for applications and libraries written in clojure and other clojure implementations.

It turned out that no clojure implementation makes it practical to integrate with system package managers. There is enough friction in playing with system-level language librareis and then turning them into linux distro system utilities that I kept using other languages like janet and python for my use cases.

I certainly can’t afford to specialize in writing a programming language.

I believe clojure can be the last language that humanity uses for high-level programming. Low-level programming can be done in languages like C and Rust and Zig. I tried to use clojure in my primary niche and failed.

That said, I think basilisp can become a system scripting language if it acquires basic project management capacity and becomes a PEP517 build backend. Jank can also become one although I wouldn’t have any expectation.

Since I don’t donate 10,000 dollars per month to any maintainer, I know that I can’t tell them what to write.

But, I just wanted people to be aware that there is an underserved niche. By telling people about this, perhaps someday, I will see a clojure implementation suitable for system scripting. I wanted clojure to be my last language. I guess I will have to live with my imperfect situation and move on from clojure anyway.

I don’t plan to turn this message into a long-term campaign. This is my last thread about this topic.

1 Like
4

Your points, @amano, resonate with me. I explored the possibility of improving Clojure’s packaging in Debian some years ago, but quickly realized it was a task far beyond my available time and ability. In fairness to Clojure as it is designed, this is a problem common to all Java-based languages, and not something that the Clojure community can unilaterally ‘solve’ without extensive cooperation with the Java community and target distributions themselves.

I would also just point out that source packages of programming language libraries aren’t usually meant to be used for development purposes - they’re typically pinned to versions useful for the end-user applications in the distribution, rather than versions that interoperate with other libraries nicely, and might lag behind the latest upstream releases by a considerable margin. I imagine you know this, @amano, but I’m just writing this here as a warning to those who might not be as familiar with system package managers!

5

TL;DR: Don’t do it.

Having maintained packages for macports publicly and maintained local builds, rpms, and debs privately for work I can tell you that no one will be happy:

  • The maintainer will not be happy selecting winners and losers from projects, plucking release candidates from upstream, maintaining multiple build and dependency trees, testing compatibility including across distros, releases, runtime versions, and possibly platforms, and fielding bug reports. That’s the exciting job of a package maintainer.
  • Users are not going to be happy. See above. If you have any success, your chosen supported distro release, versions, runtime, dependencies, etc. etc. will not be suitable for everyone. If you choose to deprecate something, you’ll discover someone depended on that. If upstream breaks something, you’ll hear it.
6

I basically gave up on packaging JVM language libraries because clojure startup time was not great for quick scripts, and it was difficult to package JVM language libraries for gentoo linux.

gnu guix system has clojure build system, ant build system, and maven build system, so gnu guix may be able to handle JVM applications, but clojure application startup time is still going to be slow. Outside gnu guix system, I think it’s practically impossible to package JVM libraries/applications for linux distributions.

If you don’t care about building from source, then you can slap an uberjar to a github release and make your package use the uberjar although clojure application startup time is still going to be slow for quick scripts. Go applications are packaged for linux distributions in a similar way.

I personally created gentoo packages for janet language libraries. I use those libraries in my janet system scripts.

I think distro language libraries can serve both applications and user scripts if the user controls one’s own package repository.

For applications that don’t require fast startup, you can use gnu guix or uberjars. But, outside JVM, there can be clojure implementations that facilitate integration with system package managers.

7

I’m curious, if you can make a package use an uberjar from a release on GitHub, why can’t you use the .tar.gz files from the release page for Babashka in the same way? e.g., Release v1.12.213 · babashka/babashka – those .tar.gz files are just the pre-built binary for each O/S & architecture.

8

Have you considered leveraging ClojureDart? It produces native, self-contained binaries with instant startup.

1 Like
9

ClojureDart runs on dart. I’m not fond of dart outside flutter GUI ecosystem.

10

Even if babashka itself becomes a binary package for a linux distro, babashka doesn’t natively support installation of a babashka library onto a linux distribution.

I haven’t tried to make babashka support linux distro package manager with classpath. In theory, I can try to write a distro package build system for babashka, but I haven’t actually tried, and I don’t know whether it’s actually possible.

A system scripting langauge can be the language that an entire linux distribution can be built on top of. For example, gnu guix system is built on top of guile scheme. Someone tried to write a linux distro on top of janet, but gave up early.

I prefer clojure as a language to janet or guile scheme, but I can’t ignore how a language implementation handles library paths and whether it can be built easily from source by a system package manager. The integration with system package manager is an essential ingredient in a system scripting language.

11

I agree to you words. I think, you explain a situation, that is inherited from Java. Java is used quite often for business applications and not so much for OS programming and scripting. But I would not give up hope. Babashka is a good step. Jank would be closer to the target architecture. I think, if the Clojure community were larger, there would someday be a unified dependency/library mechanism. You can already develop and test a library on the JVM version of Clojure and use the same jar file on Babashka. So there is already some sort of infrastructure. This is even greater than your Linux, because the same Clojure library works on the JVM, on Babashka, on Jank and in the Browser. Only at the moment there is not much interest or many resources on this, because the current Java infrastructure works well. But the last word has not been spoken. We should stay motivated.
By the way, many Python developers also prefer an own installation of Python over the version from the Linux distribution.

12

In general I don’t know if being closer to the OS is something good. Linux is based on Strings. If you pipe something from one command to another, you have ASCII text in between. That is not good, in my opinion. There is even an initiative to reimplement all the standard Linux tools in Rust. And they do the same thing: ASCII strings everywhere. That is nothing I want. I want pure data, which means, agreeing on EDN or Transit for everything. And I am happy to be platform independent. So I am not sure if Linux is a platform that is worth targeting. I think, we have more people dreaming of something like a Clojure OS than people dreaming of a native Clojure version for Linux.

2 Likes
13

I propose a step in the right direction for a system/project scripting/application language.

  • A language implementation can fetch dependencies from the user cache and the default system/OS load paths determined at the compilation time.

  • LANGUAGE_LOAD_PATH environment variable overrides the default system load paths. LANGUAGE_LOAD_PATH can be used by guix/nix profiles and also guix/nix virtual environments. Individual users should normally not set LANGUAGE_LOAD_PATH. It is generally reserved for OSes like guix and nix.

  • A load path contains each major version of each library in a separate folder.

    • /load-path/library-A/1.17.0, /load-path/library-A/2.21.0, …

    • /load-path/library-B/0.17.0, /load-path/library-B/1.21.0, …

  • Outside a project directory, the entry script “can” and “should” specify dependencies with the exact major version and the minimal minor version according to semantic version standard, and loads dependencies from the default system load paths or LANGUAGE_LOAD_PATH.

  • When inside a project directory with a project file, the project file “should” specify the exact major version and the minimal minor version of dependencies. Dependencies are fetched from the user cache, but there is an option to make the project management tool fetch dependencies from the default system load paths or LANGUAGE_LOAD_PATH.

  • The language implementation can build an uber artifact or a binary executable from a project or a script.

  • Preferably, the language implementation can be easily built by a system package manager.

It can behave like a system scripting language, a system application language, and also a project scripting language.

14

I add another feature to my proposal.

  • Multiple major versions are allowed in one process to allow slowly maintained dependencies to upgrade its dependencies later.
15

You want to build a Lisp machine. It’s been done before :slight_smile:

16

Yes, of course. Everybody wants a Lisp machine.

17

Gnu guix system is very much like a lisp machine.