How to learn the JVM well for use with Clojure?

Hello colleagues, I’m launching on a one-person (for now) “micro-startup” I will do in Clojure. I really don’t know much about the JVM, except that I have seen leaning on it with Clojure be extremely effective for some companies I have assessed as a technical diligence person, where they have done great things with small teams by just keeping everything on the JVM as much as possible, dispensing with the horror show of administering k8 and bundles of microservices, etc. So… JVM newb here, what resources do you suggest for a Clojurist to learn the dark arts of deployment and infrastructure? I’m not averse to properly learning Java in the process. (There is definitely a business advantage to being able to demonstrate interoperability with a more commodity language, after all).

thanks!

4 Likes

Regarding keeping most things in the JVM: check out one of the component libraries, like Component, Mount or others. Also be sure to use the Reloaded workflow.

Regarding web services (which I assume is what you want to do) - checkout reitit by Metosin.

Persistence: In cloud environments it’s usually prescribed to not persist to machine local filesystem, at least if you need parallell machines. For a novel take on distributed databases, check out the architecture of Datomic on-prem. It is not trivial to build a distributed database (check out Aphyr’s amazing database stress tests using his test suite Jepsen). It goes without saying that it is important to have a sound model for the data storage you need for your application.

One thing to check out closely is probably Datomic Cloud (which runs, at least primarly, in AWS). For some (not all) problems this saves you a lot of work regarding infrastructure work.

Read configuration as edn files, checkout aero.’

Use edn-files for data which changes rarely.

Checkout Fulcro and consider builing your backing with pathom3 endpoints for dataloading.

4 Likes

Most challenges of deployment and infrastructure aren’t around the JVM, but are around hosting, load balancing, SSL certificates, port forwarding, static content caching, content delivery, horizontally scaling, securing resources, managing database instances and load, etc.

While I’ve never used them personally, I think going for either Heroku, Render.com or DigitalOcean AppPlatform are probably a good way to start as a single-person startup.

Even simpler if you don’t want to load-balance is to go with a VPS like AWS Lightsail or DigitalOcean.

I think Lightsail also has a container offering that lets you do minimal load balancing.

There’s also AWS App Runnner for more advanced use cases, but I never used it, so not too sure what it does.

You could also explore going with say AWS and using ECS + Fargate along with the CDK: What is the AWS CDK? - AWS Cloud Development Kit (AWS CDK) v2 At that point you’re ready for the heaviest of use-cases, you can go full AWS and setup everything you might need with the CDK itself. It’s also possible to use EKS with this approach, but I find that even more complicated. There’s a Java API for the CDK, which you can use from Clojure.

As for “deploying” Clojure on JVM, it’s just a matter of creating an uberjar and uploading it to your host or container and doing java -jar name-of-your-standalone.jar on the host/container to run it. Assuming your host/container has java installed.

There’s also a new way with recent JVM, where you don’t even need Java installed, and instead you use jlink to even bundle the JVM alongside your app code, it uses only what it needs, so it actually ends up having smaller bundle sizes.

3 Likes

Re: deployment

For a while, we (Bloom Ventures) used Heroku (nowadays, would probably use fly.io), with hosted Postgres, and it worked fine with Clojure.

To cut costs, we got our own VPS (say, with Hetzner), and have been managing things ourselves. For a long time we would: uberjar locally (in a clean repo), scp a jar to the server, and then java -jar myapp.jar. Multiple apps on the same server. Postgres on the same server. No docker. For web apps, you’ll want a proxy like nginx in front of your app to allow for multiple domains on the server and to take care of SSL.

At some point, we threw supervisor · PyPI into the mix: it monitors the processes, restarts when they (rarely) crash, rotates log files, etc.

We use Ansible for our deploy script (I highly recommend some form of reproducible deploys, but, I don’t recommend Ansible; every complex app eventually begets a complex build, and with Ansible, you end up “programming via yaml” which is hell). Our script has evolved to build on a temporary vanilla system, run tests, and do 0-downtime deploys.

Kubernetes does a lot of nice things, which I often wish we had, and I respect the “12 factor” philosophy, but our approach has been “good enough” (and we don’t have the bandwidth to learn how to deal with kubernetes when things go wrong). Nothing stops you from deploying Clojure uberjars to kubernetes with one of the various hosted-kubernetes providers, or Heroku-clones.

Re: development

We have a philosophy of “avoid building a system for as long as possible” and try to keep things within Java-land (ex. using Hazelcast instead of Redis), within the same process (ex. bundling Lucene, rather than using a separate Elasticsearch server), and even within the same mono-repo (even if we want to run multiple seperate services, we bundle them together and launch them with different command line arguments). The main exception has been the DB (usually Postgres, but you can go quite far with H2).

As a startup, you can scale pretty far vertically on the cheap. Then as demand (and presumably revenues) increase, scale further vertically (ie. pay for a more expensive server with more CPU and RAM). Then, when that hits its limit, you can do more serious “systems design” and engineering to scale horizontally. When we went through a similar transition, the Clojure app didn’t require that many changes, most of the scaling tends to happen outside the code (ex. sharding postgres, setting up a seperate load balancer, improving the build system to now manage multiple servers, etc.)

If you’re Google/Facebook/etc., every product you release starts with Google-scale load on Day 1, so you have to do a lot of systems-design before launch. As a tiny startup, your load will start at 0, and will hopefully scale with your revenues. Too much demand is a “good problem” to have.

9 Likes

If only more people would heed this advice…

2 Likes

@rafd with multiple apps on same server do you run out of memory? Does Java just keep eating memory?

Each Clojure/Java app needs a certain amount of memory to get its work done (it depends on how many requests there are, what they’re doing, how optimized the code is, what garbage collector you use, etc.) If it doesn’t have this memory, it will crash (because it can’t allocate memory, or because it’s spending too much time garbage collecting).

On top of that, Java will use more memory, if available, to avoid garbage collection. But, it can be configured to use extra memory when available and give it back up to the OS when not needed.

So, if I run a process on, say and a 8gb RAM machine, even though that process could run with only 100mb of memory, it may still balloon to 6gb RAM usage, because the memory is available (“it’s free real estate”). But, if it’s competing with other processes, it will happily function with 100mb. (Again, the “happy” amount of memory needed depends on what the program is.)

The “use memory if available” behaviour gives Java a bad rep, because on first glance, it seems to use so much. (A similar perception issue exists with Chrome and Firefox). But, a Java process can run with much less memory if forced to. And with a lot of tuning and optimization, even less (though probably still a few multiples above perfectly-optimized C).

We’ve had a 16gig RAM server (for 20 euro/mth), hosting ~16 web apps (+ postgres dbs). Some using more resources than others, fluctuating over time. Some would grow in demand, consistently using a lot of memory, and often, the easiest solution was to move those processes to a new server (vs. spending time looking at our code to optimize for memory usage – although, proper memory leaks do need to be fixed, and sometimes, there are low-hanging fruit for memory optimization).

If you have multiple apps, let’s say 4, and a budget to support 16gb of memory, you could:
(a) have 4 apps on 4 servers each with 4gb RAM (this could also be one server w/ memory-limited VMs/containers), or
(b) have 4 apps on 1 server sharing 16gb RAM
I like to go for (b), because it gives a lot more room for any of the apps to temporarily spike in demand (and thus, usually, memory usage). In this example, any 4 of the apps could spike up to 10gb or more if needed under (b), but only up to 4gb under (a). Also, it’s often cheaper-per-gb-of-RAM to go for B than A. But, of course, there are tradeoffs: for example, it’s a greater security risk for multiple apps to share the same server; there is a also a risk that one of the apps has a memory leak and impacts other processes, etc.

5 Likes

@rafd thanks very much for the detailed responses. Your tips are very much in line with how I was thinking, and for what it’s worth, the general philosophy is very much supported by what I see in my technical diligence work (my regular job is doing tech diligence on mid stage and startups getting acquired). In fact, how well I have seen people do on a modular-but-monolithic JVM setup is one of the reasons for my adoption of Clojure!

The hidden costs of managing things like k8 with small teams is much higher than most startups realize. When I’m doing architecture consulting I always tell small teams to write it as if you will need to grow beyond small web app and single database (i.e. make sure over-coupled code won’t make this impossible) but then stick to a single relational DB for as long as you can! Transactional business logic that spans more than one data store is an order of magnitude harder.

1 Like

For anyone else who finds this thread, I had the Manning Book “The Well-Grounded Java Developer” (2022) recommended elsewhere, and it is excellent. It covers the JVM, Java build systems, and even Clojure, including concurency issues, performance tuning, and how the internals of JVM program execution works. I highly recommend it!

6 Likes

Just a small note on:

You’ll have a hard time to run any non-trivial Clojure app with just 100 MB - that is simply not realistic due to JVM overhead itself but also Clojure overhead (classloading, etc.)
The reasonable minimum I recommend is to reserve 256 MB.

You can get tiny jvm apps with java. This maven server only needs 20MB.

Are you using this technique for production too or just doing that for the dev setup? I’m also curious why you guys don’t use docker as it’s quite easy to get reproducibility with that - ie https://www.testcontainers.org

1 Like

I just started reading that book to and it is indeed great.

I’d note here also that recent versions of java have improved garbage collection considerably, so use those with your Clojure app. Also, java profilers like the one in datadog, and others work pretty well for debugging Clojure once you get used to reading the munged/post-compiled naming conventions

We haven’t felt a strong need to use Docker. Introducing Docker to our stack has tradeoffs, and the problems it solves haven’t been sufficient enough for us to make the change. (We certainly tried at some point, b/c of cargo-culting, but, then decided not to). I do recognize that Docker (or similar tools) are useful for declaring dependencies on (in our case) non-Java dependencies, like postgres databases, operating systems, imageMagick, etc., but for us, using Java-equivalents or just dealing with “read the README for which postgres version is necessary” have been sufficient for devops.

Context: our team is small (< 10), we have 5+ year developer retention (ie. we don’t have to onboard new devs all the time), we don’t do micro-services, but we do work on many independent projects.

Re: test-containers, I can see how it can be beneficial to easily declare (via Docker) and spin up a near-production system for testing. Particularly on large teams. For us, local tests with embedded modules + prod-equivalent staging servers have been “good enough” at catching errors before hitting prod.

In our prod, the things that are split-out and made into a system are usually stateful services (like a postgres database) that we wouldn’t manage in prod via a Docker container.

1 Like

this may be good for your stack:

2 Likes

A :+1: for Embedded PostgreSQL: it really helped me with testing next.jdbc since that’s such a popular DB among Clojurians, it seems.

I did eventually break down and start using Docker for testing against SQL Server and MySQL…

2 Likes

Hi there, I guess I am a bit late to the thread here, but it’s certainly an interesting topic!

I think containers or not, setting up VMs or using an app platform are both good paths. From my experience either works just fine. Mostly it’s a matter of learning either linux+systemd or docker. Then write CI for one of them.

Memory usage of the JVM is definitely a tricky topic. On shared systems it makes sense to limit the max memory available to a process. Either via JVM options or by limiting resources via docker or systemd.

A topic that hasn’t been touched here is how people debug production issues such as memory leaks and performance issues. Or simply logic errors.

Clojure has an advantage here since it has the REPL. I highly recommend enabling the REPL in production so that you can access it via SSH. This can be invaluable for debugging and even hot-fixing certain issues right in production.

To observe the production system there are many solutions for both Clojure and the JVM in general. I find it simplest to use tools not specific to the JVM.
You probably want any combination of logs+metrics+tracing. To just start out focusing on one of these might give you enough visibility into the system. Any of these can give you enough insights to setup some basic alerts and visualize behavior over time.

In my opinion the gold standard for observability these days is OpenTelemetry. The OTel standard covers traces+logs+metrics. And you can send the data anywhere you like. Java is probably the language with the most mature support for OTel at this point. I don’t have experience with using the Java OTel tooling from Clojure though.
There seems to be a promising Clojure library in the making.

OTel can be a bit overwhelming and the tooling is not yet established enough to make it trivial to get started as solo dev.
Starting with logging only is probably the simplest. But then there is the challenge to figure out how to do basic alerting and dashboarding for it.

3 simple options I am aware of:

  • If you are on AWS you could use mulog and send the structured log events to CloudWatch which allows you to do basic analytics and alerting.
  • You could use ken to instrument the application and send the data to honeycomb which has a generous free plan.
  • Grafana Cloud has a generous free tier as well and it supports metrics+logs+traces. You could start with logs and later look into metrics and/or traces. There are many ways to get logs into Grafana Loki, either from within the JVM directly or via the system logs on the host.

I hope this is useful and I am very curious to see how others approach observability and maybe other topics of operating Clojure in the JVM in production.

4 Likes

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.