Any examples of Clojure's STM being used in the wild?

I find the STM to be amazing as a concept, but I never succeeded in spotting a situation where it would provide net positive impact in the code base. Usually the applications I worked with didn’t have much state being held as native data structures, and the state that they did have could be managed with atoms just fine.

Besides, we also have core.async, which seems to be a good fit for asynchronously coordinating IO between system components with its queue-like channels. Even agents seem to be more used.

The text book use case for STM is keeping consistency when you want to coordinate changes between multiple refs in a concurrent system. I ask myself when would this be the case in a real life project. What kind of implementation would benefit from it? It seems that it only makes sense if you have state divided into multiple refs as opposed to concentrate state into few independent atoms.

So there’s that. I’d like to know which projects I could study more to try and see if spot patterns that could be useful someday. Maybe I’m not fully taking advantage of Clojure’s constructs.

4 Likes

It is an awesome concept, and barely gets used outside small examples, except I think in Haskell. I think it’s difficult to ensure good performance without a lazy language, and difficult to reason about without enforced immutability and monads.

I have a Go library for it, the performance sucks for pathological situations which come up too easily: GitHub - anacrolix/stm: Software Transactional Memory in Go.

You should look into where clojure’s STM was inspired by - which is the database.

It’s useful.

Another example is the rollback mechanisms on database-ish things such as blockchain. Rollback is really useful in many applications, especially when you can throw an error anywhere and have complete guarantee that nothing has changed.

I’ll link to http://convex.world which is a clojurish blockchain by mikera (Mike Anderson) · GitHub.

Don’t worry. I’m not being paid by them to shrill… I just really like the guy because he answered about 80% of all my questions of stack overflow when I was starting out.

Mike’s also really active on discord - Convex.

1 Like

Usually the applications I worked with didn’t have much state being held as native data structures, and the state that they did have could be managed with atoms just fine.

Me too. If you haven’t seen it yet, A History of Clojure’s section 3.6 State is a good read on this: Rich says basically the same thing :slight_smile: and discusses use cases a bit.

1 Like

Zachary Tellman outlines a relevant idiom in his book Elements of Clojure. The idiom is “If you have mutable state, use an atom”. Essentially, ref and atom accomplish the same thing in different ways. However, you’re usually better off simply combining your refs into a single atom and using swap! rather than multiple alters in a dosync.

Furthermore, databases are already really good at doing what refs do, so assuming the data you’re mutating is being persisted, why bring that into memory just to do what the database is built to do for you?

1 Like

Does the ns macro count? :slight_smile:

2 Likes

Designing with atoms results in more robust systems because of design and state is held in one place. However, there may be cases where it’s necessary to use refs but by then you’ll be dealing with other situations such as persistence because if you’re worried about distributed state, you’re probably going to be worried about what happens when the electricity goes out.

bifurcan: functional, durable data structures, also by zach, has persistence built in which is pretty cool though I’ve never used it.

I like refs for toy systems to simulate situations like locking. Also, ants.clj blew my mind with many, including myself calling it clojure’s killer app (it’s a compliment). Having said that, it depends on the application and sometimes it’s easier to do tests on the real thing.

@pmbauer, great catch. I’m liking smalltalk system more and more in terms of code versions being loaded/cached from the database, so recently I’ve been thinking about how and whys of syncing with blockchain (which serves the purpose of versioning really well) as well as a local file version.

Being completely transparent with what my thoughts are on convex lisp, I think it’s going to be hard to get people to move off the two standards right now, which bitcoin and evm. Stellar tried and failed. Algorand is doing pretty well but it’s integrating the evm. However props to Mike for giving it a go and his language is very similar to clojure, which would be great for clojure newbs to dip their toes in at a real distributed system.

I’m more in favour for developing with evm but ultimately, it not a race nor a competition and I’m going to be supporting people that do cool shit.

If I have a User and I need to transact the “balance” when say I transfer money between two users, but the rest of the User data is transactionaly independent, like updating the email or phone number.

Since I want a mutable User, I’ll use an Atom to store the User state, and I could use a Ref to store the balance inside the User atom:

(def user
  (atom
    {:email "user@user.com"
     :phone [111 222 3333]
     :balance (ref 200)}))

You can see an atom won’t work here, because we need to transact between two users, and you can’t do a swap over two atoms together, that’s why a “ref” is needed here.

We could wrap the user in a ref instead of an atom, that would work as well, but the “email” and “phone” don’t need transactions across users, only within the same user, so a “ref” isn’t needed for those and I think would be slower, though I’m not sure.

Now I also need to store all my users, and I want to be able to add/remove users. So I’m going to have a map of username → User instead of a single user:

(def users
  (atom
    {:user1
      (atom
        {:email "user@user.com"
         :phone [111 222 3333]
         :balance (ref 200)})}))

But now why keep all these inner atoms and inner refs? It even gets confusing if say you’re modifying the outer atom in one thread and the ref in another, what happens?

So you’ll often just remove them:

(def users
  (atom
    {:user1
      {:email "user@user.com"
       :phone [111 222 3333]
       :balance 200}}))

The ref isn’t needed anymore because the balance transaction would have been between two users, and now that’s a single atomic swap which handles the transaction.

In my experience this is the kind of scenario where it’s why I don’t end up using refs.

Even if you took this further, say we have Business which also has “balance” that can transfer money back and forth to Users. We need transactional support again to make sure we added and removed successfully to the User and from the Business.

(def users
  (atom
    {:user1
      {:email "user@user.com"
       :phone [111 222 3333]
       :balance 200}}))

(def businesses
  (atom
    {:business1
      {:balance 3450}}))

Here we need a solution for the transaction accross User balance and Business balance again. We could make the “balance” a ref again, or we could make the “users” and “businesses” maps a ref instead of an atom, etc.

But here too, I think quickly you figure out why not have a state map:

(def datastore
  (atom
    {:users
      {:user1
        {:email "user@user.com"
         :phone [111 222 3333]
         :balance 200}}
     :businesses
      {:business1
        {:balance 3450}}}))

Now I think there could be a performance argument. If you’re modifying this atom a lot, there’ll be a lot of contention, so maybe a design as one of the ones we had before, where the state was broken down into more vars or had some inner atoms or refs could improve performance. That said it gets tricky, refs incur more performance cost than atoms for example, so it’s hard to say which will perform best for your usage patterns.

Others have already said much the same in more detail and quite eloquently, so here’s a brief summary of my own experience. I was excited by and used STM when I was first transitioning to Clojure from Java with scattered mutable state. But as I learned better how to idiomatically use the powers of swap! combined with update and threading macros I realized I did not need it, that I was better off organizing all my state into a single atom.

It’s potentially useful where you need multiple mutable states which are related but not always updated together, and you want an optimistic locking protocol.

For example, a job scheduler might have a some data structures representing a job record, with its attendant job state (executing, pending, etc) and other bits, and then might have another set of data structure representing workers that execute jobs, worker state (e.g. health monitoring), jobs dispatched to them, and so on.

Dispatching a job to a worker means you might want to change job state and worker state in a logical transaction, and the STM will do this nicely as an alternative to a pessimistic locking model and a lock hierarchy.

A single atom might not be ideal because if there can be frequent changes to job state and worker state that are independent of the job/worker relationship.

Anyway, just an example. There are multiple ways to design these problems, but I have used the STM for a problem similar to the one I outlined (though that’s probably the only time I’ve used Clojure STM). There are also performance considerations of optimistic vs pessimistic locking which will affect design too.