Clojure should consider MoarVM as a backend

Community Center Watercooler
1

MoarVM was created for Raku language, formerly known as Perl 6.

MoarVM surpasses JVM and javascript platforms with regard to integration with linux package systems. Perl 5 already integrated well with linux packaging systems.

MoarVM has garbage collector and 6model object system. It also has JIT compiler for converting frequently used bytecode to native code. It supports parallelism and concurrency.

If clojure programmers wanted to make desktop computer applications, JVM and javascript runtime environments cannot be used because they are largely incompatible with linux packaging systems.

If clojure compiled to NQP, a low-level MoarVM language, NQP can be compiled to MoarVM bytecode and JVM bytecode. Clojure programs on MoarVM can be readily turned into linux packages.

Perhaps, someone is interested in creating a clojure implementation or a language inspired by clojure in Raku which is a good language for writing compilers.

I want moar languages on MoarVM.

moar
moar840Ă—535 522 KB

1 Like
2

If clojure programmers wanted to make desktop computer applications, JVM and javascript runtime environments cannot be used because they are largely incompatible with linux packaging systems.

That’s an inaccurate statement. There are desktop programs written in Clojure and ClojureScript. And in Java and JavaScript, for that matter. They just don’t use the Linux package management systems that often.

Apart from that, that’s not what makes Java/JS-based software incompatible with Linux packages. Because, in fact, there’s no incompatibility - it’s all compatible. There are packages in e.g. Ubuntu repositories that are Java programs that depend on other packages. Most of those packages are Java-based as well, but some are regular C/C++ libraries that Java can easily use.

5 Likes
3

Theoretically, it’s possible. In practice, it doesn’t scale up across different linux distributions because they weren’t designed to integrate with linux distributions. Their package directory layouts don’t match that of linux package systems(e.g., /usr/lib/python/site-package/…).

  1. I designed a concept of linux packages for npm modules. In straightforward cases, it’s possible by inserting symlinks into node_modules for each linux package for an npm package. But, not all npm packages are simple. I haven’t decided whether I should make a linux package for each version of an npm package or one for a range of versions. To simplify things, I should probably make a linux package for each version of an npm package. I don’t think linux distributions that install nodejs packages actually install each version of an npm package as a linux package. It would be a major burden to make and update linux packages of many versions of each package. I don’t even know how deno dependencies map to linux package versions.
  2. I designed a concept for linux JVM packages. It can be done with customized JAR build systems written by me. I cannot possibly expect package maintainers to know intricacies of JVM build systems and analyze each JVM software project’s complex build system and translate them to my customzied JVM build system. To prevent explosion of CLASSPATH, I have to create an uberjar for each JVM jar linux package. Nix, gentoo linux, and guix system may be able to rebuild JVM uberjars recursively due to advanced packaging systems, but debian and arch linux probably cannot.

Even if I somehow created clean package systems for nodejs and JVM on my linux distribution, my work cannot be automatically replicated by other linux distributions. Thus, nodejs and JVM have to come up with standard ways to integrate with linux distribution package systems.

On the other hand, perl and raku were designed to work with POSIX-like operating system package systems. Python, haskell, janet, and scheme implementations also integrate well with linux package systems. Go and Rust at least don’t make it too hard to integrate with linux package systems because they were designed from the ground up to produce static binaries.

Tell me how you can cleanly make a linux package for each npm/JVM dependency. I gave up and decided to use programming languages designed to integrate with linux distribution package systems.

If JVM forced every JVM application to upload an uberjar to maven, I may just use them in linux packages, but that’s not going to happen.

4

I don’t see how scaling and path conventions are related to be honest, but I’m not an expert here. All I wanted to mention is that your post was factually incorrect as it is possible to create desktop applications running on JVM and it is possible to have Java-based packages - and people do all those things, and more.

As an extra 5 c I can also add that I myself would never consider creating software specifically in such a way so that it could be run as a package. A good read on why: https://drewdevault.com/2021/09/27/Let-distros-do-their-job.html:

One thing you [a developer] shouldn’t do is go around asking distros to add your program to their repos. Once you ship your tarballs, your job is done. It’s the users who will go to their distro and ask for a new package. And users — do this!

1 Like
5

If your programming language is designed to integrate with POSIX-like operating system package systems, then you won’t have to ask linux distributions to add your package. They will add your softwares if they want to use your softwares.

I tried to make linux packages for electron applications and gave up. I also gave up making linux packages for JVM programs. If you actually tried doing it, you will also find it hard. I tried.

Something doesn’t scale up just because it can be done. That’s why there are not many java packages on linux distributions, but there are many haskell packages and many python packages.

My claim is that compiling clojure to MoarVM modules makes it easy to make linux packages out of clojure programs.

6

A good thing about targeting NQP is that NQP can be compiled to MoarVM bytecode and JVM bytecode.

7

I think the issue is only with Gentoo, most other Linux distro you can just bundle the app pre-built which makes it easy to package.

2 Likes
8

Gentoo linux can just put an uberjar into a package, too. But, it’s unrealistic to expect every JVM application to offer uberjars. Most JVM applications don’t offer uberjars.

There is no known way to download dependencies and then build an uberjar without internet access.

I know that I can just execute mvn install in Arch Linux package builds. But, that’s like giving up packaging. If I wanted to execute mvn install, I would skip linux packaging system.

I haven’t seen many JVM applications in arch user repository. Rather, I haven’t installed a JVM application on arch linux. I haven’t installed a single JVM application rather than JVM compilers or JVM package managers on gentoo linux.

You have to ask why there are many linux packages for python and perl applications and why there aren’t any or many JVM/javascript applications as linux packages. JVM and javascript feel cumbersome and dirty for linux packages even on debian and arch linux.

Because there is incompatibility between linux package systems and JVM and javascript, people prefer not to package JVM applications as linux packages. AsciidocFX just offers platform-specific uberjars.

The incompatibility is there, and people can feel it. If there wasn’t incompatibility issue, people would have installed multiple JVM applications, not JVM compilers or JVM build systems from linux package repositories.

When there are better alternatives such as perl and python and haskell, people don’t need to turn JVM applications into linux packages. While it’s possible to make JVM/nodejs/deno linux packages, I prefer not to because there are better alternatives that integrate with linux package systems seamlessly.

If you just want access to large ecosystems, there are perl and python and C. There are Go and Rust, too. As long as those languages let me do what I want, I don’t need to deal with JVM or javascript platforms.

My specialty is command line application. There is no need to make my work difficult with JVM or javascript platforms. For command line or terminal applications, there are better languages.

When clojure comes to MoarVM or something like it, I will consider it again for command line/terminal applications.

9

If you have an internal mirror or just a cached .m2 this is viable. I have done this on systems with no internet access. Then again…it’s simpler to just build the uberjars and bring them across the gap…

I don’t think this matters to the large market share honestly (targeting moarvm or anything). It could easily be a solo quest though if you are interested, otherwise the pitch does not seem to be compelling.

10

I don’t expect clojure developers to adopt MoarVM because of this thread. I know it is hard to create a compiler.

But, I wanted to make people conscious of the fact that if clojure developers really really wanted to target POSIX-like operating systems on desktop computers instead of web browsers or mobile devices, it should go beyond JVM and javascript platforms.

For now, MoarVM seems like the best compilation target for targeting POSIX-like operating systems.

I have a smartphone, but I rarely use it. I use my smartphone as an emergency device, not a media consumption device. I use desktop computers most of the time.

I don’t particularly love or like Raku language, but it should be okay for command line/terminal programs. For the foreseeable future, I’m going to invest in MoarVM ecosystem. I prefer Haskell, but for small terminal programs, the rigorous type system doesn’t really make much difference and slows down prototyping process.

11

I don’t see many Python or Perl applications packaged for Linux either. The only difference is most distros come with Python and Perl installed, so you’ll see more use of it for scripting, but still I don’t see many applications that are really in Python or Perl.

And Uberjars are exactly what you want. In fact, since Java 11 you should even make a self-contained package that even bundles the VM with it, that’s why they introduced modules in the first place.

I don’t see what’s wrong with that. Maven is not supposed to be a package manager for applications, you’re not supposed to use it to install Java apps, it’s meant as a build tool to bring in dependencies when developing, not as a user to install Java apps. That’s why Uberjars were made, so you can bundle your Java app in one file and provide it with a launcher that relies on an existing JRE or tries to install one if missing.

And now, like I said, you should use jpackage instead to make a self-contained bundle that doesn’t even need a JRE.

This applies to Clojure as well.

So if I were making a desktop application in Clojure, I’d most likely use jpackage, or go the Uberjars+launcher route, or even go GraalVM native compilation, and I would distribute that to my users.

Same goes for a command line, I don’t expect a command line app to pull down a million dependent files personally, here too I’d rather something self-contained. And that’s what I’d do if I intended it to be used by Linux users, as opposed to Clojure developers.

12

At least part of the problem with trying to package up Clojure libraries and apps for Linux is that the policies of the distro typically want the packaging process to build everything from source code in isolation (i.e., the build must not reach out and pull “random” prebuilt stuff down from the Internet).

I’ve seen various people pop up on Slack over the last year or two, asking about trying to do this for various Linux distros and running into problems. One of the issues, for example, is that to build Clojure from source now, you need a version of clojure.spec.alpha available as a library and to build that from source, you need a version of Clojure itself. And pretty much anything built with Clojure these days needs Spec (this came up in a discussion about packaging the Clojure CLI for Linux, which means that tools.deps.alpha needs to be packaged, which means Spec, which means Clojure, etc).

To build Clojure from source in this way, you need to bootstrap a version that predates Spec being a separate library (so Clojure 1.8 although maybe early prerelease builds of 1.9 would work), then use that to build Spec and core.specs and then use those to build Clojure 1.10, and then use all that to build t.d.a. (along with building from source all the Aether stuff – Java source – that it also needs). I’m not sure how targeting a different VM reduces the pain of that.

The whole build-everything-from-source policy is pretty much incompatible with the Java ecosystem because so much Java software assumes Maven is the source of pre-built JARs for dependencies, and with Clojure you add Clojars as well, and it becomes pretty much untenable. I imagine this is why it’s also incompatible with JS/Node/npm.

For the vast majority of Clojure (and Java) users, the Linux packaging process is irrelevant – they’re building apps from dependencies on Clojars (and Maven) and they’re not looking to distribute those apps via Linux’ package management.

The vast majority of the application target market is macOS/Windows and even when folks do want to target Linux as well, they’re just going to bundle their uberjar in some sort of simple executable or installer and avoid the pain of the standard package managers.

As a complete aside, re: smartphone usage – at work, our users are consumers and we’ve watched browser shares switch over to majority mobile and in some markets, desktop browsers are now barely even a blip on the scale. In some of our markets, the smartphone is the only “computer” the vast majority of our users own.

3 Likes
13

Smartphones and tablets are for media consumption. It’s mostly for docile consumers.
I think not having or using a desktop computer is a mistake. Smartphones are turning people into docile consumers who don’t have a voice on the internet. The trend set by big tech is to not even show multiple search results but just give one answer through voice AIs. When the ruling class just chooses one right answer for each topic, people can be programmed by the ruling class.

The ruling class is trying to eliminate the ability to have voice on the internet and search for multiple perspectives about topics. They want to silence people’s voice and eliminate multiple perspectives and present only the mainstream take on everything through voice AIs.

Are you going to create an alternative media website on smartphone or tablet? No. You aren’t going to edit videos and create video interviews on mobile devices. You aren’t going to write in-depth articles and do server maintenance on limited mobile devices.

And, you should make a leap to linux or BSD because windows and mac os are increasingly getting more surveillance capabilities. So, I develop for my niche which is POSIX(linux) desktop computers.

I’m a producer. I produce stuff on desktop computers. Thus, I strongly prefer linux package system to uberjar. Some people use apple tablets for some of their production, but desktop computers are more flexible and more powerful in terms of production. Without keyboard and mouse, production becomes difficult.

It’s convenient to install python/haskell/perl/raku libraries as linux packages and use them in python/haskell/perl/raku interpreters without worrying about creating a project directory.

When a script gets big, I create a module/project directory.

You can’t do this with maven and uberjars. System scripts are nice convenient tools for producers. As long as producers exist, system scripts are relevant.

There is a malware element to it, too. Before packages are built in linux repositories, they are vetted by linux distribution maintainers. When there is no person who vets builds, you get malwares. Malwares have been found on npm registry.

14

While I understand that clojure developers don’t care much about creating tools for desktop linux, developers who target POSIX operating systems on desktop computers aren’t going to use clojure.

If clojure developers want to really target POSIX operating systems on desktop computers, look into creating another compiler that targets NQP which is compiled to MoarVM and JVM. I don’t yet know another virtual machine suitable for POSIX desktop computers.

15

You are very paranoid… :slight_smile:

If anything, the consumerization of the Internet – through smartphones and tablets – has created an explosion of perspectives and voices: everyone can be a content producer now. Just look at TikTok (as awful as I think it is, it has truly provided a platform for the masses).

Yup, people are already doing this. As I said, in some countries, smartphones are ubiquitous and desktop computers are rare, and those countries still produce a lot of content online.

I know quite a few people who use speech-to-text on smartphones to write a lot of content (a friend of mine has used speed-to-text almost exclusively for all his computer access for decades due to serious RSI preventing him from typing).

I’ve been doing server maintenance via my smartphone for years. It’s not my primary choice of device for that, but with VPN apps and ssh terminal apps, it’s very tractable to do this.

That is an increasingly niche group targeting an increasingly niche platform – and if that’s your target, that’s fine, but your needs are going to be very different to the vast majority of developers – and numbers/market share is what drives where money and effort is spent on platform support.

4 Likes
16

I think you’re really on point. This has been at the back of my mind for a while. It’s all the more unfortunate that there is no Clojure on Android! (outside of making JS/web apps).

The startup time “issue” is a bit overblown and it feels like a chicken/egg thing - there is no interest b/c it doesn’t exist. The past year in Clojure has had some exciting stuff going on. Recent stuff like scicloj have made Clojure branch out of the major money maker of backend web tech - but even the recent GUI libs (HumbleUI + CLJFX) eshew mobile.

Does anyone have a good sense of what are the technical challenges with running Clojure on ART? I’d like to poke at this but I don’t want to go down the rabbit hole without having a broad sense of the problems :slight_smile:

17

Are you including GitHub - vouch-opensource/krell: Simple ClojureScript React Native Tooling in this? It seems like you get cljs directly on react native and ios/android from there. https://cljsrn.org/ has a wider scope of stuff too.

18

I hadn’t actually seen that project :slight_smile:
Definitely looks interesting and I get why people go the JS route, b/c of the iOS support.

I feel I’m finally kinda groking the JVM stack and I guess I’m just not super eager to dive into the JS ecosystem (which is even more turtles-all-the-way-down) - so I’ve never looked at the JS solutions seriously.

Project idea #1 was to write a small AR app I have in mind. Doing that in JS when you start with a JVM lang seems a bit strange to say the least… though there are computer vision libraries in JS so it’s not totally insane… but it sounds masochistic

The other thing I kinda looked at is porting a GUI app I have to mobile - but that uses JVM libs so that’s also a no-go.

I wonder why they don’t have a demo APK on the krell webpage

19
  1. I don’t code for money. I make money from other things. Why do you have to get paid for coding? I code for other developers and other linux/BSD users. I am an open source developer.
  2. Python is hugely popular, and it is also largely used on desktop/server market. Desktop market isn’t a small niche in development community because developers use desktops. Developers need to develop for their own tools and their own environments.
  3. JVM doesn’t reach much beyond android and web backends. If you are writing web backends or iOS apps, you don’t have to use JVM. I always didn’t like JVM module system.
  4. I see a bright future for MoarVM. Yesterday, I learned that Raku not only integrates well with POSIX operating systems but also has probably the most advanced module system among all programming languages. Raku allows developers to import the same modules written by different authors and different versions of the same module. Other languages targeting MoarVM can use the same module system. I think android should replace its own version of JVM with MoarVM.

No. I mean in-depth alternative media websites that provide alternative narratives and alternative perspectives. Not shallow TikTok entertainers who don’t care about narratives. Not YouTube game streamers. Narratives and perspectives are world control mechanisms. If you are deliberately fed only one perspective and you don’t even have the ability to search for alternative perspectives, you can be easily programmed or numbed or controlled by the ruling class who control most media outlets. It’s very important to be conscious of narratives. You don’t run an alternative media platform on TikTok. If you want an alternative media, you get your own website. You aren’t going to edit your videos professionally on limited mobile devices like smartphones.

1 Like
20

@catdog I think you raise some interesting points and some of them resonate with me personally in some ways.

If you believe in MoarVM so much, and you are raising this point on a clojure community site, have you considered porting clojure to MoarVM yourself?

There are a number of “ports” of clojure that might be interesting because it shows that porting clojure is feasable:

1 Like