@didibus have you heard of plumatic/plumbing and keechma-pipelines (cljs only)?
plumbing won’t solve the error handling but, as far as I understood, it offers a way to resolve dependencies of functions within a graph so you would end up with a linear execution after the first call. In contrast, keechma-pipelines creates a linear pipeline of functions and manages the runtime state for you.
Checking whether the email is present in the system at the start of validation isn’t enough though. There’s a race condition where multiple requests could try to claim the same email: both could pass that validation and then one will succeed, inserting the email into the database, and the other will likely fail, assuming email is a unique key in your database – and you’ll get an exception anyway.
I don’t think you can just wrap up the logic in some “nice” monadic construct and be done with it. You’re going to get exceptions in the real world. You can try to wrap up the exceptions as well but you’re going to need to either use try/catch anyway or reinvent that construct somehow wrapped up in your monads 
I mean the Type itself, in the sense of having a wrapping structure around the return.
Like with invite-user-to-project!, in Clojure you could easily have that function return different things, maybe it returns true if succeeded, nil if missing user, :no-project-found if missing the project, etc.
Though I’ve done things like wrap in a vector or a map often, but that’s also a lot less ceremony than creating a custom result type.
I’ve also seen this in the past and thought about it for some cases: GitHub - lambdaisland/uniontypes: Union Types (ADTs, sum types) built on clojure.spec but never really felt the need for it I guess.
Hadn’t seen keechma/pipeline before, but I had seen plumbing, though never tried it. I think my issue at first glance with both of them is that they seem a little limiting still.
Plumbing I don’t see how you’d setup conditional and recursive flows. With let you can do:
(let
[foo (get-something)
bar (case foo :buzz (do-buzz) :bazz (do-bazz)
biz (save foo bar)]
{:result biz})
Which eventually I’ve found I need to do some conditionals in my flows at some point.
Or for recursive flows like:
(let
[foo
(loop [foo (get-something) attempt 3]
(if (and (nil? foo) (pos? attempt))
(recur (get-something) (dec attempt))
foo))
bar (case foo :buzz (do-buzz) :bazz (do-bazz)
biz (save foo bar)]
{:result biz})
And for keechma/pipeline it seems that you can’t choose prior results to pass to the next, it’s always the previous result passed to the next function. So say you want to do:
(let
[thing (get-thing)
other-thing (get-other-thing)
foo (do-something thing other-thing)
baz (save foo)]
{:result baz})
I guess you can maneuver around it either by using the pipeline state or by threading a map, not sure if there’s downsides to that.
In general, I’ve thought about this kind of thing before, I’m sure there could be something ergonomic, but it’s a balance between using standard Turing complete Clojure to model whatever flow you want, or come up with something a bit more structured/frameworky that lets you do it possibly more conveniently and declaratively with maybe some nice features like custom scheduling of steps and all that, but at the detriment of maybe being more limiting in some ways, or simply being more cryptic and requiring people to first learn the “framework”.
In a single threaded non-blocking context like ClojureScript I can see more value in such a thing, since using normal Clojure to describe these asynchronous flows do require a lot more work and the code can get unruly. But for threaded blocking contexts like Clojure I’m not sure it’s worth it.
That said, I think it’s a matter of experimenting and maybe someone will find something that has really nice ergonomics and isn’t limiting and plays nice with the rest of Clojure.
Another approach to this is, instead of defining what I call a “flow”, or a “workflow”, which I consider a “pipeline” a subset of. You can instead model a state machine.
So like instead of saying do this, then that, then depending on do this or that (like with let). Or instead of saying do this but require that to happen first, which requires that other thing (like with plumbing). What you would do is do something and return a transition and feed that to your state-machine. I haven’t tried it, but it be an alternative.
Off-topic: I’ve noticed some posts were withdrawn from the thread (post withdrawn by author, will be automatically deleted in 24 hours unless flagged) including one of my own. I haven’t withdrawn any of them, and I’ve sent a message at #clojureverse-ops in Slack to check what’s happening.
Sorry if one of your posts is flagged.
My choice has been to use cond-let
(cond-let
(invalid-input? input)
(err :invalid-input)
:let [data (read-data somewhere)]
(nil? data)
(err :no-data-at-somewhere)
:let [ok? (protect-business-rules data)]
(false? ok?)
(error :protected-rules-failed)
:let [data' (transform data)
ok? (save-data! data')]
(not ok?)
(err :failed-saving-data)
:else
:ok)I’m reposting the gist link – thanks @pieterbreed for adding the failjure example 
I’ve created a Gist with a problem example and different implementations for it based on what has been discussed here (it’s working in progress).
https://gist.github.com/wcalderipe/a117b1a2058a5a2910f8eee160ad8d7e
Feel free to leave your comment there as well.
That’s cool, and another viable option, I would like to see a variant that is like better-loop where you can also recurse to arbitrary locations in the loop.
I don’t understand, please expand, what is this better-loop?
After thinking about this for a second; here is what I pasted in the gist referenced in this part of the thread.
(defn validate-email
[{:as user-record
:keys [email]}]
(if (re-matches #".+@.+\..+" email)
user-record
(f/fail ["Invalid e-mail format" {:email email}])))
(defn find-db-user-by-email
[email]
(case email
"alice@mail.com" {:id "df19e9e0-20a4-4aa8-9e89-320a5edc1950"
:name "Alice"
:email "alice@mail.com"}
"bob@mail.com" (throw (ex-info "Error establishing a connection to the database." {}))
nil))
(defn validate-not-taken
[{:as user-record
:keys [email]}]
(let [r (find-db-user-by-email email)]
(if-not r user-record
(f/fail ["Email already in use. {}"]))))
(defn save-user!
[user-record]
(assoc user-record :id (str (UUID/randomUUID))))
(defn create-user!
[user-record]
(try
(let [result (f/-> user-record
(validate-email)
(validate-not-taken)
(save-user!))]
(if (f/failed? result)
{:status 400
:body (f/message result)}
{:status 200
:body "OK"}))
(catch Exception e
(error e "Exception while attempting to create new user")
throw e)))
Sorry, I realize I really didn’t explain myself and was imagining something not really obvious.
I am referring to GitHub - Engelberg/better-cond: A version of cond that supports :let clauses, and a number of other conveniences. and was thinking that I’d like something like this that could also let you loop inside it. I’m not sure it makes total sense, but I’m picturing something like:
(better/cond
:let [valid-data (validate-data input)]
(nil? valid-data)
{:result :invalid-data}
:loop :retry-get-user [user (get-user (:username valid-data)) attempts 3]
(and (= :error user) (pos? attempts))
(recur-to :retry-get-user (get-user (:username valid-data)) (dec attempts))
:loop :retry-get-user-project [project (get-project user) attempts 3]
(and (= :error project) (pos? attempts))
(recur-to :retry-get-user-project (get-project (get-project user) (dec attempts))
(nil? project)
(recur-to :retry-get-user (get-user (:othername valid-data) 3)
:else
{:result project})
I created a library which I use in production for 2 projects GitHub - fmnoise/flow: Functional (and opinionated) errors handling in Clojure
Readme has pretty much everything but in a few words it utilizes instances of ex-data(or any other Throwable) as error container, while everything else is assumed to be non-error, and has some functions as well as let like macro for building flow using that convention. Works perfectly for my needs, zero-dependencies, no new abstraction invented (despite there’s a possibility to do that using supplied protocol), pretty much easy to combine with traditional exception throwing approach and bit more tricky but also doable to combine with Either-style stuff.
Thanks for this question, it sparked a very interesting discussion. I’m not going to add much to the discussion, but just throw in a small interesting technicality that might leave a bad taste in your mouth:
user> (def nine 9)
;; => #'user/nine
user> (:pipeline/error nine)
;; => nil
user> (def kw :keyword)
;; => #'user/kw
user> (:pipeline/error kw)
;; => nil
user> (def some-error {:pipeline/error "this didn't work"})
;; => #'user/some-error
user> (:pipeline/error some-error)
;; => "this didn't work"
So you could build pipelines where you check for the presence of an :pipeline/error key, and abort if so.
This is basically what GitHub - fmnoise/anomalies-tools: Anomalies handling tools has done.
I found this thread really interesting and enlightening. I recently took some old code and made it into a library which lives in this space (GitHub - kardan/taxa: An experiment in hierarchical domain logic in Clojure and ClojureScript.). It’s essentially control flow using variants, a registry and some utilities. Some of the projects mentioned here sounds really similar.
Several of the solutions offered here feel very monadic and/or “type-heavy” and do not feel idiomatic to me for use with Clojure. Like several posters in this thread, I’ve also explored the idea of having a pipeline that “understands” success/failure modes and conditionally does the right thing. I made a library of it, years ago, but after using it at work we decided that it just wasn’t a nice way to write Clojure in larger-scale applications and we refactored away from it (I linked to it in passing above but I’m including a more descriptive link this time – and I’ll lift some of the readme into this thread so folks don’t actually have to click through and read about why I archived it):
Some of Engine’s ideas were too abstract to work well but in particular:
In addition, the core concept of running an “Engine request” through your entire business logic produced monadic code that was very hard to read and non-idiomatic, from a Clojure point of view. We’ve recently rewritten that code to use “native” Clojure to query data sources and access hash maps etc, and to create a simple pipeline of closures to be executed on success. The result is much simpler, more idiomatic code (which could still stand a bit more simplification but is already an improvement on Engine-based code).
So where we ended up was building a queue of thunks – no-arg functions – to be executed in order on completion of a pipeline of operations. It’s stateful and dynamic: we use a dynamic var holding an atom wrapping a (persistent) queue to hold the “execute on success” thunks and swap! conj to add each thunk. It turned out to be simple and pragmatic and allows us to treat the “completion” as a consistent transaction.
The monadic code looks good “in the small” but in complex business logic it just didn’t scale well. Similarly, our solution looks terrible “in the small” but turns out to be simple and surprisingly elegant for complex business logic.
I agree, I think the current “problem” is too simple. What we want is a way to describe a flowchart with code.
We need to tackle a few more complexities I think:
Conditional branching on more than errors. Sure, an error happens, you want to just short-circuit the whole flow and return an error. But what if it isn’t an error, but instead it’s a lookup of a config or the computation of some value that dictates the decision of where to go next?
Looping, this is often needed in more advanced use cases of implementing a complex process. And here I mean looping back to prior steps so they are done over again, possibly with more data or changed data going to them at that point.
Handling errors at different levels. It’s only a matter of time you’ll need to actually handle some of the errors you get, you won’t always just short-circuit the whole flow and return to the user an error. You’ll want to handle the errors, and you might need to handle them at various levels. Imagine a set of transactions and you need to add a Saga pattern to undo the prior changes if any of the later changes failed for example, and then imagine once recovered you want to retry the whole thing again in case it was a transient issue.
Actually getting an exception thrown by the runtime or something not under your control, and needing to handle it or short-circuit on it.
Reuse of parts of the flowchart in others. Eventually you’ll see that chunks of a given flow exist in other flows, and you might want to share those across the board so changes to all of them can be made more easily, and so they remain consistent.
Composition of the steps into new flows. Ideally, you don’t want the way you’re doing things to couple the steps together, steps shouldn’t have to care what happened before and what’s happening after, they shouldn’t need to know anything related to the flow, otherwise they’ll be hard to compose and reuse to build other flows. But you also don’t want that to make it hard to wire up a new flowchart using your existing steps.
Debugging, it’s only a matter of time you get an error or a bug where things don’t work, and you need to figure out where and how things broke down to fix it. How easy is it to introspect and identify the point of failure and what caused it?
Idempotency of some steps or skipping over parts of the flow. When the process involves none repeatable operations, say sending an email, and say you have retries, or errors thrown after and the user retry the whole operation, but you don’t want two emails sent out? Imagine sending the email is itself 5 steps composed together in a flow, maybe you want to skip over that, or you want to make that sub-flow idempotent on its own, etc.
Maybe we could come up with an actual smaller example that needed all of these? I think that be a much better demonstration of the real life challenges that could be encountered when dealing with real control flow on an API implementation which models a business process.
That’s actually an approach proposed by Cognitect in GitHub - cognitect-labs/anomalies and I just built a toolset around it. The idea isn’t bad but as for me it’s bit weird that some map could represent error while other map couldn’t. It’s basically the same as Either build on maps/records, so ex-info with data containing error details feels way better as error container as for me. And that’s the reason why I archived that repo and switched to flow.
I wonder why no one has mentioned interceptors nor middlewares here on this thread. Recently, a colleague has done a cool experiment with metosin/sieppari to splitting business procedure into a pipeline with short-circuits between some of the steps. Have anyone used interceptors to build something alike?
I understand that monadic approaches may not feel idiomatic to Clojure. However, I wonder what about promises on ClojureScript? Since we’ve stepped on that landscape, we’re haunted by monads because lots of functions in JavaScript nowadays will return a promise.
@jumar check @mjmeintjes’ post on this gist for a simple example using missionary.
@fmnoise nice library you have out there. Thanks for sharing it. Feel free to add some examples on this gist, if you want. 
@seancorfield I’m having difficulties picturing what you’re trying to say and how does the queue o thunks looks like. At least for me, it’s hard to visualize abstract concepts without some examples. That said, I would love to read how does the queue solution look like and what’s idiomatic Clojure code is to solve business logic pipelines for you. Would you mind sharing some snippets with us here (they don’t have to follow the problem proposed on the thread if you don’t want to)?
@didibus thanks for sharing the specs of a complex pipeline. I’ll try to get a new example diagram covering some of your points and will share it here soon.
I don’t have anything outside the context of work code – which is proprietary and can’t be shared – but the concept is pretty simple: as you work through the (pure) business logic you (swap! *actions* conj #(some-func :args)) for “stuff” that needs to get done, in order, if the entire threaded pipeline succeeds. Then at the end of the (pure) business logic, if you succeeded, you “commit” those actions by doseq'ing over the queue and calling each thunk.
For context, this grew out of code that originally used a monadic library that “forced” all external “reads” out to the front and all external “writes” to the end – Engine – so our focus was on lifting all side-effecting code out of the pipeline to the end, which goes back to my comment about the small examples in this thread being far too simple to illustrate the problems in the real world (and why I appreciated @didibus 's post about all the edge cases that need to be considered). Essentially, use the full range of Clojure’s control structures, use exceptions, separate logic from side effects (by pushing the side effects out to the edges – which we also see in imperative shell, functional core).
Thus sounds very similar to how missionary works. You first setup the graph of tasks to run, and then you execute the graph. Each task is just a function that takes a success and failure callback, and returns a cancel function (although missionary abstracts this away so you don’t deal with the functions directly). If one task fails, the entire graph is cancelled.