How are clojurians handling control flow on their projects?

We often have to write imperative logic in our programs for wiring pure and side-effectful procedures as shown in the image below.

image

Throwing exceptions is something most of us are familiar with because of inherited habits from OOP and is the first approach that comes to mind. However, I’ve been looking for a data approaches (more elegant?) to solve the control flow problem.

How are clojurians handling cases like that in your projects with Clojure(Script)?

Other directed questions I have are:

  1. How to pass data from one step to another in the pipeline taking into account side-effectful procedures?
  2. How to control the flow of the pipeline? For example, if the “Protect business rules” procedure fails, how to abort without throwing exceptions?
  3. Any libraries you would recommend?

Thanks :raised_hands:

9 Likes

I’ve had a lot of joy employing failjure for use-cases exactly like this. It has macros that correspond with ->, ->>, cond-> and let (and more…) and this makes it dead simple to have a flow of data exactly like in your diagram. It works by short cirtuiting on “failures” so you get back either the first/earliest failure/validation-error OR you the transformed data in a state you can rely on.

5 Likes

I don’t really see the point in avoiding throwing exceptions - it is well supported in Clojure and works well. But I’m interested to hear other perspectives.

I’ve have started to use missionary to glue together side-effectful procedures.

4 Likes

Exceptions don’t really have anything to do with OOP so I’m not sure why you think they’re bad because of that?

Exceptions are perfectly idiomatic in Clojure (and ClojureScript) but, in my opinion, should be used for “exceptional” situations – things you either don’t expect or can’t handle – and not for flow of control (and they shouldn’t be used for flow of control in any language, OO or not, as far as I’m concerned).

So what are you left with? You’ll mostly see either some sort of conditional branching or a monadic approach where the error data and success data are carried along together. Some people like the latter but I generally don’t, except in limited situations where the traditional conditional branching would just be too distracting.

For the example above, my decisions would be based around what kinds of “failure” each step could produce and how (or even “if”) it should be reported to the caller. Some things to consider are i18n/l10n if failures have to be reported to humans, whether something like GitHub - cognitect-labs/anomalies might be appropriate, whether data needs to be conveyed back to the caller in terms of a failure, and so on.

For “validate input”, it might be acceptable to just throw an exception if the function has no sensible behavior when given invalid input – or perhaps you need to communicate to the caller what was wrong at some level.

For the “data read”, perhaps it will throw an exception (which you probably can’t handle either) or perhaps it might return nil/() and you need to take action based on that (which may also involve communicating failure to the caller).

The same applies to the business rules: how can they fail, what can you do about those failures, what should you communicate to the caller about those failures.

And so on.

Every situation is likely to be different so I would expect lots of different approaches depending on the answers to all of the above.

10 Likes

Would you mind elaborating on your use of missionary? Glancing through the GitHub project pages I have a hard time imagining why I would want to use it.

I feel sometimes it’s hard to make a decision whether I should throw an exception or return empty for the function I’m writing. It feels like it depends on the callers. So what should I do in this case?

1 Like

Have a look in this thread: How to replace DI in Clojure?
Lots of valuable insight on the topic.

1 Like

My rule of thumb for this is: is the “error” expected or not (and is an empty result “expected” or not)?

I would say that in any case where the program can continue with “no result” indicating the function didn’t find/compute anything, that’s the right choice (and an exception is not).

On the other hand, if the condition is “unexpected” and your function cannot handle/recover from it, then throwing an exception is acceptable (and it’s up to the caller to deal with it or not).

Consider “reading some data”: if the data should always be available (modulo network connectivity etc), then throwing an exception if no data is read/found/computed makes sense – that’s an “unexpected” condition; if the data may or may not be present, then returning nil makes sense – the possible lack of data is “expected”.

5 Likes

That’s a great question! I’m interested in answers here as well.

I use a modified let macro called trylet which goes like this:

(with-metrics
  (trylet
    [valid-input (validate-input input)
     read-data (perform-read valid-input)
     rules-passes (protect-business-rules valid-input read- data)
     transformed-data (transform-data read-data (:opts valid-input))
     saved (save-data transformed-data)]
      ;; Response is created and returned here
      {:result :success}
    (catch Exception e
      (log/error
        (str "Failed in API foo with details: "
          {:valid-input valid-input
           :read-data read-data
           :rules-passes rules-passes
           :transformed-data transformed-data
           :saved saved}))
        ;; Error response is created and returned here
        {:result :error})))

What the trylet macro does is that it makes available the bindings inside the catch as well. Where they’ll be nil if they haven’t been bound yet, because the exception was thrown before reaching them.

So I model the data-flow in the let part, which I’ve found is better then a threading macro, because then each step is named (more readable in future) and you can easily do things like take two prior return to pass to the next step or choose where the return of a prior step must go (like first arity, second, thirds). So it’s the most flexible. It also allows you to model more complex flows, like with conditionals, which is not as well supported with threading.

And if anything throws an exception, it short-circuits (unless I explicitly try/catch it inside the let binding), but in my catch because all the bindings are available, I can see what succeeded what failed and what hadn’t run yet, so it’s very easy after to debug.

Edit: Oh, also I would normally move the validation outside of it, as I don’t consider it a part of the business logic. So really things are setup in two:

(defn api-foo
  [input]
  ;; Validate
  (if-not (valid-input input)
    {:result :invalid-input}
    perform-foo))

Where perform-foo is the above function with the trylet.

That’s because I like my business logic to be able to assume they got valid input. It means I can test them with a generative spec. It’s one less thing to have in them.

In api-foo I would also have all the scaffolding for the particular server I’m hooked up on, so this is a bit of the facade pattern for those familiar with that. So that means in api-foo I would validate and conform and coerce all the input into the exact shape and type my business logic wants, and I would unwrap the business logic result and similarly convert it to whatever response is needed for the given server.

6 Likes

That’s a very interesting approach – is that macro available in an OSS library somewhere?

Unfortunately not :frowning: at this moment.

In my own group at work I’ve seen 4 independently inspired attempts to use this kind of short-circuiting result+error chain pattern: Vavr Try in Java code and err->> plus 2 homegrown Clojure attempts, one of them mine. One of these was painfully removed and the others painfully remain.

The idea is not wrong. It’s beautiful, simple™ and completely robust. It seeks to achieve composability for side effectful steps. Done well those steps can be built and tested as ordinary functions, and “mapped” or “lifted” by other 2nd order functions lift(f) → F that wire ordinary f(x) → y functions that could throw err to chainable functions F([x err]) → [y err] that passes through both if an err was passed in. What’s not to like?

In practice the costs are high. It forces programmers to design for reuse prematurely (often.) It makes them fit your composition model in an all-or-nothing way that’s difficult to get out of. It requires discipline to keep the chain and the “lifting” occurring in one place, otherwise the [result err] tuple starts crossing layers/abstractions and complicating otherwise straightforward functions. It also precludes getting information X and using it 3 steps down without passing X through every step in between (not so bad in Clojure where flowing a huge context map through steps is very flexible and idiomatic, but very hard in Java!). After coming across articles like these I gave up trying to sell it.

My particular foray into this was for pulling together DRY (reusable) steps for cross-service HTTP integration tests. Although state-flow looks interesting I’m now espousing persona driven DAMP (no reuse) tests built using large let blocks for similar reasons to @didibus above - easy peasy. We just don’t need anything harder.

Ultimately I think I would only recommend this approach where large numbers of steps had to be composed dynamically. Or perhaps where your domain requires fallback patterns - like returning pre-defined recommendations instead of ones customised for you when that’s too slow or otherwise fails in a shopping cart app. Then this approach beats catching exceptions I think :slight_smile:

15 Likes

That “Against ROP | F#” article in particular is really good (all three are good and just underline why I don’t like monadic code in Clojure – and why I archived my Engine project!).

2 Likes

That is indeed an interesting article, from the same person popularizing the concept in the first place, awesome!

I think all this result stuff is pretty over engineered in Clojure. A lot of things from static languages are over engineered in my opinion when used in Clojure, and result types are one of them.

Clojure can easily model any function to return different type of things in different circumstances, you do not need a custom common type of closed possible values.

Also, remember some of the benefits of “openness” is software evolution over time. There is nothing more annoying then updating the version of some dependency only to now have your code fail to compile because it added one more option to its closed error result type which you’re not handling.

Instead if you had a default, then even if they added more in the future, you’re not broken.

Wow I also love this one linked from one of the others: No Best Practices - Satisfice, Inc.

I’ve been telling people this forever, new team members with less experience normally hate me when I say “all best practices are average at best :joy:

Edit: I even have a little informal proof to convince them after their hesitation…

If this practice is best, and it is known to be, it is only a matter of time untill it is common for code bases to put this practice into use. Thus it is only a matter of time until most code bases make use of it, at which point that it becomes a common practice it can no longer be best, where your average code base will now make use of it, there will be another better code base with a different practice. Thus it is important for us to go beyond the best practices!

2 Likes

Thanks for the insights on the thread, folks. I’ve got some good leads already.

@mjmeintjes I’m not avoiding them. In fact, we’re using exceptions in lots of places. I’m just wondering what else is out there.

About missionary, I’ve heard of it recently but I didn’t have the time to play with it yet. I’m curious to see what the results of your experiments are. Would you mind sharing what specifically have you tried to solve and how did you with us here?

@seancorfield don’t get me wrong, I never said exceptions are bad – sorry if the text wasn’t explicit enough.

What you’ve said about “validate input” and “data read” are aligned with what I already have in mind. That’s a good sign!

Let’s say we have to create a user and validate if the e-mail is already taken. What I have in mind are two different steps:

  1. Find an user by e-mail. If not found, return nil (data read)
  2. If not nil, that means the e-mail is already taken (protect the business rule)

In that context, the caller should know what “nil” means and translate it to a proper message.

A side note about returning nil. Using nil for the absence of data is a good option for contextualized procedures only, in my opinion. For example, if we have a function named find-user-by-email!, returning nil clearly means “user not found”. However, having a more “complex” pipeline like invite-user-to-project!, returning nil doesn’t say much to the caller about the “why” and “what” of the failure.

@didibus after researching a bit here in the ClojureVerse and GitHub, I’ve noticed every project has its own sauce for handling situations like that. So, I’m very curious to see what people have been doing on their projects that are outside those spotlights, what was the rationale behind their decisions to use whatever they’re using now, and how they feel about it.

Your answer summarizes well the points mentioned above, thanks for sharing. My previous project used to have a similar approach to your trylet. Would you mind share a gist of the macro in this thread?

@Anthony_Leonard thanks for sharing your experience and the articles – they are all good.

I’ve heard of ROP and that Elixir uses it a lot but I never saw anything that got traction on the Clojure community; hence, this post for exploring other people’s minds.

@didibus do you mean result types or result schemas?

I know Elixir uses pattern matching for control flow over result maps. I find the approach very interesting. However, most examples I’ve read weren’t on top of complex and side-effectful pipelines. I think core.match is an attempt to bring that onto Clojure land but I never heard of anyone using it. If someone knows a project or has experience with it, I’m interested to read what are your thoughts on it.

2 Likes

@didibus have you heard of plumatic/plumbing and keechma-pipelines (cljs only)?

plumbing won’t solve the error handling but, as far as I understood, it offers a way to resolve dependencies of functions within a graph so you would end up with a linear execution after the first call. In contrast, keechma-pipelines creates a linear pipeline of functions and manages the runtime state for you.

Checking whether the email is present in the system at the start of validation isn’t enough though. There’s a race condition where multiple requests could try to claim the same email: both could pass that validation and then one will succeed, inserting the email into the database, and the other will likely fail, assuming email is a unique key in your database – and you’ll get an exception anyway.

I don’t think you can just wrap up the logic in some “nice” monadic construct and be done with it. You’re going to get exceptions in the real world. You can try to wrap up the exceptions as well but you’re going to need to either use try/catch anyway or reinvent that construct somehow wrapped up in your monads :slight_smile:

3 Likes

I mean the Type itself, in the sense of having a wrapping structure around the return.

Like with invite-user-to-project!, in Clojure you could easily have that function return different things, maybe it returns true if succeeded, nil if missing user, :no-project-found if missing the project, etc.

Though I’ve done things like wrap in a vector or a map often, but that’s also a lot less ceremony than creating a custom result type.

I’ve also seen this in the past and thought about it for some cases: GitHub - lambdaisland/uniontypes: Union Types (ADTs, sum types) built on clojure.spec but never really felt the need for it I guess.

Hadn’t seen keechma/pipeline before, but I had seen plumbing, though never tried it. I think my issue at first glance with both of them is that they seem a little limiting still.

Plumbing I don’t see how you’d setup conditional and recursive flows. With let you can do:

(let
  [foo (get-something)
   bar (case foo :buzz (do-buzz) :bazz (do-bazz)
   biz (save foo bar)]
  {:result biz})

Which eventually I’ve found I need to do some conditionals in my flows at some point.

Or for recursive flows like:

(let
  [foo
   (loop [foo (get-something) attempt 3]
     (if (and (nil? foo) (pos? attempt))
       (recur (get-something) (dec attempt))
       foo))
   bar (case foo :buzz (do-buzz) :bazz (do-bazz)
   biz (save foo bar)]
  {:result biz})

And for keechma/pipeline it seems that you can’t choose prior results to pass to the next, it’s always the previous result passed to the next function. So say you want to do:

(let
  [thing (get-thing)
   other-thing (get-other-thing)
   foo (do-something thing other-thing)
   baz (save foo)]
  {:result baz})

I guess you can maneuver around it either by using the pipeline state or by threading a map, not sure if there’s downsides to that.

In general, I’ve thought about this kind of thing before, I’m sure there could be something ergonomic, but it’s a balance between using standard Turing complete Clojure to model whatever flow you want, or come up with something a bit more structured/frameworky that lets you do it possibly more conveniently and declaratively with maybe some nice features like custom scheduling of steps and all that, but at the detriment of maybe being more limiting in some ways, or simply being more cryptic and requiring people to first learn the “framework”.

In a single threaded non-blocking context like ClojureScript I can see more value in such a thing, since using normal Clojure to describe these asynchronous flows do require a lot more work and the code can get unruly. But for threaded blocking contexts like Clojure I’m not sure it’s worth it.

That said, I think it’s a matter of experimenting and maybe someone will find something that has really nice ergonomics and isn’t limiting and plays nice with the rest of Clojure.

Another approach to this is, instead of defining what I call a “flow”, or a “workflow”, which I consider a “pipeline” a subset of. You can instead model a state machine.

So like instead of saying do this, then that, then depending on do this or that (like with let). Or instead of saying do this but require that to happen first, which requires that other thing (like with plumbing). What you would do is do something and return a transition and feed that to your state-machine. I haven’t tried it, but it be an alternative.

1 Like

Off-topic: I’ve noticed some posts were withdrawn from the thread (post withdrawn by author, will be automatically deleted in 24 hours unless flagged) including one of my own. I haven’t withdrawn any of them, and I’ve sent a message at #clojureverse-ops in Slack to check what’s happening.

Sorry if one of your posts is flagged.