How are clojurians handling control flow on their projects?

Hi! I have made several approaches to this problem and I use the next solution.

1 Like

I created a library which I use in production for 2 projects GitHub - fmnoise/flow: Functional (and opinionated) errors handling in Clojure
Readme has pretty much everything but in a few words it utilizes instances of ex-data(or any other Throwable) as error container, while everything else is assumed to be non-error, and has some functions as well as let like macro for building flow using that convention. Works perfectly for my needs, zero-dependencies, no new abstraction invented (despite there’s a possibility to do that using supplied protocol), pretty much easy to combine with traditional exception throwing approach and bit more tricky but also doable to combine with Either-style stuff.

1 Like

Thanks for this question, it sparked a very interesting discussion. I’m not going to add much to the discussion, but just throw in a small interesting technicality that might leave a bad taste in your mouth:

user> (def nine 9)
;; => #'user/nine
user> (:pipeline/error nine)
;; => nil
user> (def kw :keyword)
;; => #'user/kw
user> (:pipeline/error kw)
;; => nil
user> (def some-error {:pipeline/error "this didn't work"})
;; => #'user/some-error
user> (:pipeline/error some-error)
;; => "this didn't work"

So you could build pipelines where you check for the presence of an :pipeline/error key, and abort if so.

This is basically what GitHub - fmnoise/anomalies-tools: Anomalies handling tools has done.

1 Like

I found this thread really interesting and enlightening. I recently took some old code and made it into a library which lives in this space (GitHub - kardan/taxa: An experiment in hierarchical domain logic in Clojure and ClojureScript.). It’s essentially control flow using variants, a registry and some utilities. Some of the projects mentioned here sounds really similar.

Several of the solutions offered here feel very monadic and/or “type-heavy” and do not feel idiomatic to me for use with Clojure. Like several posters in this thread, I’ve also explored the idea of having a pipeline that “understands” success/failure modes and conditionally does the right thing. I made a library of it, years ago, but after using it at work we decided that it just wasn’t a nice way to write Clojure in larger-scale applications and we refactored away from it (I linked to it in passing above but I’m including a more descriptive link this time – and I’ll lift some of the readme into this thread so folks don’t actually have to click through and read about why I archived it):

seancorfield/engine: A Clojure library to implement a query → logic → updates workflow, to separate persistence updates from business logic, to improve testing etc.

Some of Engine’s ideas were too abstract to work well but in particular:

In addition, the core concept of running an “Engine request” through your entire business logic produced monadic code that was very hard to read and non-idiomatic, from a Clojure point of view. We’ve recently rewritten that code to use “native” Clojure to query data sources and access hash maps etc, and to create a simple pipeline of closures to be executed on success. The result is much simpler, more idiomatic code (which could still stand a bit more simplification but is already an improvement on Engine-based code).

So where we ended up was building a queue of thunks – no-arg functions – to be executed in order on completion of a pipeline of operations. It’s stateful and dynamic: we use a dynamic var holding an atom wrapping a (persistent) queue to hold the “execute on success” thunks and swap! conj to add each thunk. It turned out to be simple and pragmatic and allows us to treat the “completion” as a consistent transaction.

The monadic code looks good “in the small” but in complex business logic it just didn’t scale well. Similarly, our solution looks terrible “in the small” but turns out to be simple and surprisingly elegant for complex business logic.

3 Likes

I agree, I think the current “problem” is too simple. What we want is a way to describe a flowchart with code.

We need to tackle a few more complexities I think:

  1. Conditional branching on more than errors. Sure, an error happens, you want to just short-circuit the whole flow and return an error. But what if it isn’t an error, but instead it’s a lookup of a config or the computation of some value that dictates the decision of where to go next?

  2. Looping, this is often needed in more advanced use cases of implementing a complex process. And here I mean looping back to prior steps so they are done over again, possibly with more data or changed data going to them at that point.

  3. Handling errors at different levels. It’s only a matter of time you’ll need to actually handle some of the errors you get, you won’t always just short-circuit the whole flow and return to the user an error. You’ll want to handle the errors, and you might need to handle them at various levels. Imagine a set of transactions and you need to add a Saga pattern to undo the prior changes if any of the later changes failed for example, and then imagine once recovered you want to retry the whole thing again in case it was a transient issue.

  4. Actually getting an exception thrown by the runtime or something not under your control, and needing to handle it or short-circuit on it.

  5. Reuse of parts of the flowchart in others. Eventually you’ll see that chunks of a given flow exist in other flows, and you might want to share those across the board so changes to all of them can be made more easily, and so they remain consistent.

  6. Composition of the steps into new flows. Ideally, you don’t want the way you’re doing things to couple the steps together, steps shouldn’t have to care what happened before and what’s happening after, they shouldn’t need to know anything related to the flow, otherwise they’ll be hard to compose and reuse to build other flows. But you also don’t want that to make it hard to wire up a new flowchart using your existing steps.

  7. Debugging, it’s only a matter of time you get an error or a bug where things don’t work, and you need to figure out where and how things broke down to fix it. How easy is it to introspect and identify the point of failure and what caused it?

  8. Idempotency of some steps or skipping over parts of the flow. When the process involves none repeatable operations, say sending an email, and say you have retries, or errors thrown after and the user retry the whole operation, but you don’t want two emails sent out? Imagine sending the email is itself 5 steps composed together in a flow, maybe you want to skip over that, or you want to make that sub-flow idempotent on its own, etc.

Maybe we could come up with an actual smaller example that needed all of these? I think that be a much better demonstration of the real life challenges that could be encountered when dealing with real control flow on an API implementation which models a business process.

6 Likes

That’s actually an approach proposed by Cognitect in GitHub - cognitect-labs/anomalies and I just built a toolset around it. The idea isn’t bad but as for me it’s bit weird that some map could represent error while other map couldn’t. It’s basically the same as Either build on maps/records, so ex-info with data containing error details feels way better as error container as for me. And that’s the reason why I archived that repo and switched to flow.

I wonder why no one has mentioned interceptors nor middlewares here on this thread. Recently, a colleague has done a cool experiment with metosin/sieppari to splitting business procedure into a pipeline with short-circuits between some of the steps. Have anyone used interceptors to build something alike?

I understand that monadic approaches may not feel idiomatic to Clojure. However, I wonder what about promises on ClojureScript? Since we’ve stepped on that landscape, we’re haunted by monads because lots of functions in JavaScript nowadays will return a promise.


@jumar check @mjmeintjespost on this gist for a simple example using missionary.

@fmnoise nice library you have out there. Thanks for sharing it. Feel free to add some examples on this gist, if you want. :smiley:

@seancorfield I’m having difficulties picturing what you’re trying to say and how does the queue o thunks looks like. At least for me, it’s hard to visualize abstract concepts without some examples. That said, I would love to read how does the queue solution look like and what’s idiomatic Clojure code is to solve business logic pipelines for you. Would you mind sharing some snippets with us here (they don’t have to follow the problem proposed on the thread if you don’t want to)?

@didibus thanks for sharing the specs of a complex pipeline. I’ll try to get a new example diagram covering some of your points and will share it here soon. :raised_hands:

1 Like

I don’t have anything outside the context of work code – which is proprietary and can’t be shared – but the concept is pretty simple: as you work through the (pure) business logic you (swap! *actions* conj #(some-func :args)) for “stuff” that needs to get done, in order, if the entire threaded pipeline succeeds. Then at the end of the (pure) business logic, if you succeeded, you “commit” those actions by doseq’ing over the queue and calling each thunk.

For context, this grew out of code that originally used a monadic library that “forced” all external “reads” out to the front and all external “writes” to the end – Engine – so our focus was on lifting all side-effecting code out of the pipeline to the end, which goes back to my comment about the small examples in this thread being far too simple to illustrate the problems in the real world (and why I appreciated @didibus 's post about all the edge cases that need to be considered). Essentially, use the full range of Clojure’s control structures, use exceptions, separate logic from side effects (by pushing the side effects out to the edges – which we also see in imperative shell, functional core).

2 Likes

Thus sounds very similar to how missionary works. You first setup the graph of tasks to run, and then you execute the graph. Each task is just a function that takes a success and failure callback, and returns a cancel function (although missionary abstracts this away so you don’t deal with the functions directly). If one task fails, the entire graph is cancelled.

1 Like

I looked at Missionary after I saw in mentioned in this thread and it doesn’t appear to be anything like either Engine or what we are now doing at work (and I have to say the example code in the Missionary repo and in the Gist posted here is impenetrable as far as I’m concerned: short, cryptic names that don’t tell me what they do, and bizarre syntax).

Alright Sean I was trying to stay out of this thread but you pulled it out of me. You’re right that Missionary needs better documentation and is hard to learn in its current state, so let me elaborate about how important it is. Missionary is very low level concurrency primitive for referentially transparent IO actions, RT dicrete streams and RT continuous signals. You might think of missionary as a reactive virtual machine with reactive assembly instructions. Missionary is a competitor to core.async, but with a much more modern functional programming approach that has benefitted from studying advances in functional effect systems, for example Scala’s ZIO, as well as reactive systems like Jane Street’s Incremental. Missionary is implemented with metaprogramming, not monads, but can express monadic control flow for IO actions, streams and signals. Missionary has made it clear that pure functional programming and reactive dataflow programming unify, they are the same thing.

You’ll be interested in hyperfiddle/photon, a new library from team Hyperfiddle that I teased on twitter, which is a reactive and distributed dialect of Clojure/Script positioned for web development. Imagine React.js but full stack: incremental view maintenance all the way from database views to DOM views, as one streaming computation.

Photon implements a custom clojure analyzer in a macro (like core.async) to compile Clojure syntax to an abstract DAG which is then executed in reactive fashion by missionary. Ironically, we implement try/catch syntax on purpose, compiling it back into missionary operations to get reactive try catch (as well as reactive if, reactive control flow, reactive for, reactive fn, etc). Photon has distributed, reactive closures. You can intuit this as distributed mapreduce distributing your reactive program across the client/server distributed system, serializing the dynamic environment and streaming it on the fly. It’s really fast, and it is way better at web client/server datasync than hand-coded IO, and it composes properly. Distributed client/server expressions where the database flows directly to the view, and with the full, undamaged composition power of lisp.

The problem with try/catch in Clojure is that it interacts terribly with lazy sequences, which are total macro hackery and these hacks break equational reasoning in really unfortunate ways. To fix this, the language itself must be lazy. Which is the case with Photon.

7 Likes

The syntax takes some getting used to, but once you have used it for a while it makes a lot of sense. Probably similar to the core.async syntax.

I’ll try to explain why I think the approaches seem similar (keeping in mind that I’m just a beginner at missionary):

From how I understand your approach, you would have a atom of thunks that will get executed at the end of your business logic - *actions*. Each of the thunks is just a no-arg function that then gets serially executed at the end of your business logic (via doseq).

For example:

(defn business-logic [actions]
  *some pure business logic*
  (swap! actions conj #(some-func1 :args))
  *some more business logic*
  (swap! actions conj #(some-func2 :args)))

(comment
  (business-logic *actions*) ;; pure
  (doseq [a *actions*] (a) ;; side-effectful
)

The core of what missionary provides is the task abstraction - where a task is just defined as a function that takes a success and failure callback, and returns a no-args cancel fn. Missionary provides an interface on top of this abstraction to make it very simple to compose tasks into a graph via normal clojure code.

So, for example, using missionary to implement your approach (as far as I understand your approach), I would simply add

(defn business-logic []
  (m/sp
    *some pure business logic*
    (m/? (m/sp (some-func1 :args))
    *some more business logic*
    (m/? (m/sp (some-func1 :args))))

(comment
  (def task (business-logic)) ;; pure
  (m/? task) ;; execute task while blocking, side-effectful, cancel using normal thread interruption
  ;; OR, for non-blocking execution
  (def cancel (task #(println "SUCCESS" %) #(println "ERROR" %))) ;; cancel by calling (cancel)

The difference with the second approach is that your business logic can use the results from tasks (for example, some-func1 might be a task that calls an external api and returns something that you need for subsequent steps.

2 Likes

If you wanted to replicate your approach more fully, you could do:

(defn business-logic [actions]
  *some pure business logic*
  (swap! actions conj (m/sp (some-func1 :args)))
  *some more business logic*
  (swap! actions conj (m/sp (some-func2 :args))))

(comment
  (business-logic *actions*) ;; pure
  (def task (apply m/join vector @*actions*)) ;; returns a task that runs all the tasks concurrently (but all on the same thread in this case)
  (m/? task) ;; execute and run the tasks, returns the result of executing each sub-task in a vector
)

At least that confirms it is nothing directly like Engine or what we’re currently doing – so I gleaned that amount from the docs, correctly! – although @mjmeintjes goes on to show how to achieve a similar thing (to what we’re currently doing) using Missionary which I certainly would not have gotten from docs/repo.

Missionary sounds fascinating, now that you’ve elaborated on what it is intended to do, and I’m certainly interested in alternatives to core.async :slight_smile: Photon also sounds fascinating so I’ll have to put that on my “reading list” when you release it.

Based on this, I’ll have a play with Missionary. Thank you!

1 Like

Are exceptions not slow in Java.

I’m coming from .NET background, and using exceptions for flow control would be a big NO! Exceptions are incredibly slow.

for example, on my machine, checking if a file exists 1million times v trying to read file and see if it throws exception.

Checking if it exists takes 14seconds
FileNotFoundException 32 seconds

Parsing 1 million integers 00:00:00.0026347
Parsing 1 million integers where it might throw an exception: 00:00:05.1844392

Exceptions make a stack trace, unwind the stack etc. This is a slow operation.

I’m uncomfortable using exceptions / try catch in clojure because of my experience from .NET of how slow exceptions are, but maybe I shouldn’t be?

Exceptions are designed for “exceptional” situations and shouldn’t be used for regular “flow of control” (as I noted above).

If you “expect” a file to be missing, use .exists() on the File object. If the file being missing means that you can’t continue, throw an exception.

If you “expect” incoming data to be parsable as integers, just use Long/parseLong and let it throw an exception if you get bad data. If you “expect” to get some bad data and you can do something about it (such leaving it as a string value or converting it to zero), then maybe it’s worth doing some check on the input to avoid having to try/catch around Long/parseLong.

There are ways to construct Exception objects without the overhead of the stack trace etc but if you’re not (ab)using exceptions for “flow of control”, that shouldn’t be necessary.

With the Clojure CLI and -X option, for example, the way to have a function cause clojure to exit with a non-zero status is specifically to throw an exception: that says “I failed! I can tell you why but I can’t do anything about it!” so they definitely have their place.

2 Likes

Well, I just saw this interesting topic - lots of great ideas here!

So, most of my personal projects are ClojureScript, and I tend to use promesa. Probably, if you’re using ClojureScript, most (if not all) your side-effects functions will be async in some way, and promesa integrates really well with Javascript’s promises, so that’s my “to-go” library. It also handles errors beautifully, so that’s another plus.

On the other hand, if I do have lots of inputs in a function that can come from side-effects (like read data from a database, then other piece from HTTP, then something else) I would use pathom. It also handles errors in an interesting way, and even better, on pathom3 you can define multiple “paths” from your data (so if something fails, it’ll try another path). But that’s just for “resolving data”, not by “saving you data in multiple places” or “provoking multiple mutations”.

2 Likes

@wcalderipe Okay I tried to come up with a more complex example, here’s the gist for it: Example of a complex business process to implement in Clojure. Please link to your solutions for alternative way to implement the same in Clojure (or other languages). · GitHub

I think it be interesting to see what are different ways to implement that same example in Clojure (or even in other languages).

@dustingetz For ClojureScript that looks really interesting, I’ll keep an eye out for it.

@mjmeintjes Using Missionary for control flow is an interesting angle, but if you don’t have async requirements, would it still be a good way to do it? Do you feel up to giving it a try with my example, and re-write it using Missionary instead?

4 Likes

Two approaches I’m interested in which have similar characteristics involve reifying the data flow in the program in some way.
One approach which also handles concurrency is using core.async pipelines. It’s also pretty simple to build a DAG representation which can be compiled to a running system.
I don’t think I’ve seen solutions tackling this approach yet, but when I ran it by colleagues they said it feels hard to conceptualize. Could be because it connects what (the function to execute) with how (which pipeline, etc.). It’s also pretty noisy to have to consider backpressure, and splits in the data flow make it hard to track.
Its counterpart is sort of inverted, which is to use state machines.
If the transition between states is defined by a pair of functions, one to get the next state and the other emits effects, we can build a pure, reactive system. It feels like it has a lot in common with the ideas Dustin mentioned.
A state machine can accurately represent the flow of data in the system and completely separates concerns of how/when from what. We can build elaborate and efficient execution models on top of it.
This idea is still rather unformed but I wonder how far it can be taken. Can an entire application be built on top of it?
edit: This definitely ties to @kumarshantanu’s call to action to build better machines. We still haven’t found the right level of abstraction and language to describe them.

2 Likes