Organizing code to minimize side effects and mocking

[depdev]

Hello Clojureverse! First time posting

I’ve been checking out at Clojure for a while by watching a bunch of Clojure talks, and am just starting to familiarize myself with the language & ecosystem & idioms.
Given that, I wouldn’t be able to use Clojure at $DAYJOB, although I’m eager to be able to apply all the learnings/patterns/principles/idioms that still fits my current project.

Coming from the amazing Solving Problems the Clojure Way talk by Rafal Dittwald, I want to focus on organizing my application code. I know this is a very common topic (DDD, Clean architecture, Functional core imperative shell, etc) but I’m intrigued by the concepts discussed in the talk: minimize, concentrate, defer, and I believe the concepts can be applied in most languages (just differ in the comfort level as different language provides different constructs). This question might not be Clojure specific, but I was introduced to this from Clojure and want to know how Clojure devs approach this topic.

What I want to achieve:

1. I want my app to consist of a preferably thin layer of impurity (state: db conn, http client, cache), perhaps in the outermost layer of the app, in my case the HTTP handlers/controller.
2. I want the “core” or the biz logic to be pure, so I can easily test with very minimal mocking needed.

Issues I have:
A. How to handle dependencies for the pure “core”/biz logic?
When the biz logic needs to fetch something from DB, check some property of the returned value, and do an HTTP request with the returned value, do I pass the DB conn/HTTP client? That will require me to mock. If I pass in just the value and let the controller call the DB, then my controller have to know what data to pass to the biz logic. Even if I call the DB in the controller, I still need to do HTTP call afterwards, if I return a “result” object, then my controller have to know what to do with the result. This feels like I’m splitting the biz logic within the controller and the pure “core”.

B. How to reuse pure “core” biz logic functions?
For example, in a RSS Reader application, once a user is registered, I want to create some default “collections” to store the RSS feed items (“Saved stories”, “Unread stories”). My biz logic then would need to:

1. check if user email already registered (func IsEmailInUse(email string) bool)
2. hash password (func GeneratePasswordHash(pass string) string)
3. insert user (func InsertUser(user User) error)
4. insert default collections (func InsertDefaultCollections(userID int) error)

each of the action above are impure as I need to interact with DB and they depend on the previous steps. Relating to problem A, if I have a pure function for each of the steps, problem A will just become harder to solve as I will need to repeatedly pass dependencies, or pass a function, or do the steps on my impure layer (controller).

Hoping I can get some advice on how to approach some of the issues, how people usually approach these things in Clojure, or just to discuss this in general!
I’m open to learning more Clojure features/libraries if the question is best answered in Clojure, after all I’m learning the language too!

Thank you!

[Linus_Ericsson]

I think your example is great! It is a small, simple and common flow, that quickly gets really complicated. I cannot consume the video right now, but the captions mentions transactions (as in transitions between consistent states), which is a key item in all this.

To try implement this, let’s go through the different functions in the flow you mention.

IsEmailInUse - the only way to answer this would be to query all the email adresses stored in the system (preferably indexed in some way).

GeneratePasswordHash - you claim this is impure - yes, but it has also other important properties that helps us here (this is highly implementation specific) but essentially you can run this function “anywhere” as long as you know the password to hash, and the result, before it is persisted, is not that important (you can generate a new hash if the previous one was lost).

InsertUser needs to know if the user is already in the db.

InsertDefaultCollections must know if the collections for the user is already created to be able to do the right thing.

The most naive way to solve this problem would be to:

1. Lock the DB for all changes (and reads) but your own.
2. Do the look ups nescessary to know what operations that really needs to be run.
3. Apply the changes.
4. Unlock the DB.

This is only possible in a single threaded system. When the database is on the other side of a vast “network ocean” (in terms of time to answer) and the system is undeterministic (the network can be down etc) and the process can crash and leave transactions open - locking the whole system.

Also, what happens if there is an error and the system crashes while in the transaction lock, etc etc (Kyle Kingsbury, aka Aphyr has made some serious testing of these things in things that claims to handle distributed systems well, and found serious flaws in many of them).

To gain reasonable performance we need to apply changes while other things are happening in the database. The major problem is the changes that could happen between the look ups and the application of changes (the transaction). But how?

We need to be able to read state and then create a transaction instruction. When applied, the transaction should be applied atomically, all or nothing, to guarantee the system to be in a consistent state. In practice this means that if there where a change in the database that affected our transaction, the transaction should fail, for instance there was already an email in the system when we was about to create that same one. When there is a break of consistency, we have to generate a new transaction instruction which takes in data from the state the system is in now.

If we ignore the external side effects for now, and only looks into the persistence layer, what would a possible solution be?

For IsEmailInUse we could have some index in the process (a hashset or even a bloom filter) that made it possible to question if a certain email was availiable in the state we know. This could be cached in the application, but doesn’t have to, as long as we have can ask if an email exists or not and the know that if this situation changes, the transaction will fail and we can retry it later, eventually be correct.

For the transaction instructions to work correctly, the update statement must be able to essentially compare and swap to a previously empty email when applied. Please note that this is a less extensive requirement than to look all the emails in the database up to see if this one was already taken.

The hash of password can be precalculated and inserted, but fail together with the rest of the transaction if there is a breach in the other requirements which is enough for us as the flow is specified now.

The user - can be handled as with the email - if the preparing logic found that the user does not exist, it should be created, under the consistency check that the user is not already created when applying the transaction (this could be invalidated by the compare and swap of the email, and perhaps other things).

Default Collections - same thing - are there default collections. If the collections already exists, the instruction to create them should be omitted from the transaction instruction.

Essentally you add an extra step, a compilation of constraints if you want, that creates update instructions that can either succeed or fail and thereby cancelling the whole transaction.

With quite some work this can be extended to work over multiple transactions (sometimes called a “saga”). The saga handling should be able to recover from system crashes (as should the one-transaction flow described above do).

Side effects could actually be seen as similar to the transactions in the saga pattern, but with even more complicated retry logics.

OK, how to create this?

We need to be able to compile transaction rules that are quick to apply but still has atomic guarantee. The entity requesting the transaction must be made aware of the result (succees/fail) of the transaction to be able to make retries etc.

The most reasonable way to do this is to have some kind of transaction/change log, to be able to keep indexes (of any structure) up to date in our application server (this is what Datomic do!) and we need sane ways to query the data in unforeseen ways (to manually add various indexes as the need appears becomes unmaintainable).

Regarding mocking/testing/clean core. Datomic has an in-memory database which is very, very useful for testing and makes it easy to compile transaction instructions, which can be used to run the logics of the application.

To chain several transactions, this is called “Saga pattern” and is mentioned shortly in Tim Ewalds video on Reifying time in Datomic.

Side effects (that are not part of the transaction mechanisms/repeatable reads from database): You will have to represent the state of the side effect in an explicit way. You will need to figure out how to handle all the cases (including time outs/no answer) for the side effects and the transaction instruction compilation process must be able to handle al the possible cases.

There can be a need for rollbacks if, for instance we have a ledger system somewhere, and later fail a saga, then the already issued ledgers has to be reversed.

That’s all there is to it!

Unfortunately very few distributed applications models these things exhaustively and carefully enough, IMO. Distributed systems with consistency guarantees are hard to get right, for all the wrong reasons.

[didibus]

For a good example on your topic, check this example out: GitHub - didibus/clj-ddd-example: An example implementation of Domain Driven Design in Clojure.

A. How to handle dependencies for the pure “core”/biz logic?
When the biz logic needs to fetch something from DB, check some property of the returned value, and do an HTTP request with the returned value, do I pass the DB conn/HTTP client? That will require me to mock. If I pass in just the value and let the controller call the DB, then my controller have to know what data to pass to the biz logic. Even if I call the DB in the controller, I still need to do HTTP call afterwards, if I return a “result” object, then my controller have to know what to do with the result. This feels like I’m splitting the biz logic within the controller and the pure “core”.

You would fetch the data the business logic needs from the DB in your controller, pass it to the pure business logic, and it would return a “result” object, and you’d then make an HTTP call based on that result from your controller.

This does not split the business logic, the problem is you are thinking of the application logic as the business logic. Side-effects are part of your application logic, which in your example can be managed by your controller. And business logic is also part of the application logic, but it’s the part related to the rules and constraints that govern changes to the state of your application.

You can think of an application as a program that maintains state, and given commands, transforms that state and optionally performs actions (like sending emails, showing something on screen, play music, etc.)

For each command, a controller is called. The controller models what the command is meant to do. But it will leverage the pure business logic to decide if/how to transform the application state, and what actions to perform, if any.

B. How to reuse pure “core” biz logic functions?
For example, in a RSS Reader application, once a user is registered, I want to create some default “collections” to store the RSS feed items (“Saved stories”, “Unread stories”). My biz logic then would need to:

1. check if user email already registered (func IsEmailInUse(email string) bool)
2. hash password (func GeneratePasswordHash(pass string) string)
3. insert user (func InsertUser(user User) error)
4. insert default collections (func InsertDefaultCollections(userID int) error)

You’re describing logic that is mostly side-effect/actions, so it’s normal you feel most of this would just happen in the controller, and it should.

If I look at your example, where business logic exist is probably inside InsertUser. That can be business logic.

Remove the actual inserting into the DB from the InsertUser function. Instead, have that function return if a user should be inserted into the DB or not, and what the user information to insert should be.

func InsertUser(isEmailInUse bool, email string, password string, other-user-info ...)
  if isEmailInUse
    return [error, "User already exist"]
  else
    return [userInsertedEvent, new User(email, hash(password), other-user-info...)]

Now your controller first runs a query on the DB to get the isEmailInUse value. Then it calls the pure InsertUser business logic. If that returns an error, the controller in turn displays an error back to the user. If it returns a userInsertedEvent instead, the controller should now run a query on the DB to insert the user from the userInsertedEvent. I’m assuming the DB insert returns the next auto-increment ID for that user, so now the controller has the userId as well, and now once again it should call the InsertDefaultCollection but that too should be a pure business logic.

func InsertDefaultCollections(userId string, user User)
  if user.country == US
    return [defaultCollectionInsertedEvent, userId, new DefaultCollection(...)]
  else
    return [error, only US default collection supported]

And now once again, the controller would either display the error, or make a call to the DB to insert the default collection for userId.

Controller would look a bit like:

func CreateUserController(email string, password string,  other-user-info...)
  db.startTransaction
  isEmailInUse = db.isEmailExist(email, forUpdate = true)

  userInsertedEvent = InsertUser(isEmailInUse, email, password, other-user-info...)
  if error(userInsertedEvent)
    db.rollback
    return view.handleUserInsertedError(userInsertedEvent)
  else
    db.handleUserInsertedEvent(userInsertedEvent)
  userId = db.getLastInsertId()

  defaultCollectionsInsertedEvent = InsertDefaultCollections(userId, userInsertedEvent.getUser)
  if error(defaultCollectionsInsertedEvent)
    db.rollback
    return view.handleDefaultCollectionsInsertedError(defaultCollectionsInsertedEvent)
  else
    db.handleDefaultCollectionsInsertedEvent(defaultCollectionsInsertedEvent)

  db.commit

The controller implements the full application logic for the CreateUser command. But inside the controller, all it does is orchestrate between the DB, the View and the BusinessLogic. The role of the controller is to orchestrate what needs to happen for the full application of the CreateUser command.

The db’s role is to offer queries over the DB. In my example, the DB actually knows how to map domain events from the business logic to the queries needed for it on the DB, which is why it has those “handleDomainEvent” methods.

The view’s role is to generate the HTML to return, again here I made it aware of domain events returned from the business logic, so it knows what HTML to render for a given domain event or error.

The business logic’s role is to determine if and how to change the state, if the state needs to be changed, it returns a domain event describing the change to make, if it’s illegal to change the state in the requested way (such as because the email is taken already), it returns an error saying that the state cannot be changed.

[depdev]

Thanks for such a comprehensive answer! I learned a lot from this.

Yup for that example I would also think to wrap everything in a transaction, so that it becomes an all-or-nothing action, since it would fit the requirement.

Transaction instructions seems interesting, am I correct to imagine this as a data structure that contain commands to be executable (or like a function to execute)? So in terms of testing, I can split this into 2 parts: pure function that returns the transaction instructions/command, and impure layer that calls that pure function and apply the dependencies of that instruction?

func HandleUserRegistration(inputs) {
    txn = startTransaction()

    instructionOrError = UserRegistrationInstruction(inputs)
    if instructionOrError.error then return errorPage(error)

    instructionOrError.action.apply(txn)
    txn.commit()
    return successPage()
}

func UserRegistrationInstruction(inputs) {
  // check if email is in use
  // not sure how should i model this, if this requires a query to the DB, but access to DB is not within this function (just on the caller)
  if emailInUse return {error: "email in use"}

  // hash password, construct the user-record-to-insert
  if error return {error: "some error"}
  stmt1 = "sql stmt to insert user here"

  // check if collection already exists, if not construct the collection-record-to-insert
  if collectionAlreadyExists return {error: "some error"}
  stmt2 = "sql stmt to insert collections"

  return {
    action: txn -> txn.exec(stmt1, stmt2)
  }
}

Or perhaps I can split func UserRegistrationInstruction to its components and let func HandleUserRegistration call and handle each of them

Thanks for mentioning the in-memory DB, that made me think it’ll be better to have a DB to run the tests against. Not that I don’t want to run tests against DB, but from past experience it’ll be either: make test runs longer since I need to run DB containers (via testcontainers, but in-mem DBs will be faster for sure, then this issue is no longer valid), or I need to mock the DB conn, HTTP API clients, cache conn, etc everywhere its being called throughout the code, which when there’s a lot of them it’s easy to forget what to mock. That’s why I’m now trying to explore a way to minimize impure code and have much more pure code, so that testing is straightforward. (Experience coming from working on Java & Golang projects)

Thanks for mentioning concepts like sagas, Datomic. Although I might not have the immediate need to use them, the concepts are new and interesting for me

[depdev]

Thanks for the explanation and examples! This clears some of my unknowns.

First, I just realized indeed I was confused between business logic and application logic, as you mentioned.
My understanding was that business logic also includes fetching the required data to do the business logic (deciding something, inserting something, removing something), and the rest (HTTP controller, Kafka processor) will just validate and translate the incoming request and the outgoing response accordingly.

Returning events/errors and having the controller orchestrates these events/errors makes sense to me, and it allows easy testing of the individual components (the pure biz logic, the domain-aware DB/repository and view layer). Maybe to go further, I can mock all the components of the controller and the test cases might be easier to write (to isolate, just to test the if-elses, error handlings, etc within the controller code, without having to construct the needed values for the biz logic/view/db). I’ll definitely try this out.

After checking out the repository you mentioned, would it be correct to call the controller as the application service? One that have access to infra (db, cache, http api) as well as orchestrating the biz logic function calls. If so, is it natural for the application service to grow more than the rest of the codebase, in terms of:

• having to handle errors
• if-elses to handle different outcomes/events
• initializing db/cache/api clients
• usage of test doubles (db stubs/mocks)
  and for most cases, the application service is the place to check out first in order to understand the general flow of the application?

I still have to digest the example repo and familiarize myself more with DDD terms, but I’m starting to get the gist of it, at least in terms of separating the responsibilities (code for pure business logic, code for orchestrating outside-world-state & pure business logic calls).

Thanks for sharing this, it was helpful for me!

[didibus]

@depdev

After checking out the repository you mentioned, would it be correct to call the controller as the application service?

Yes, in an MVC framework that would be the controllers.

Though some people choose to make the controller a facade to separate application service(s), to isolate it from the specific concerns of the MVC framework you’re using.

@depdev

My understanding was that business logic also includes fetching the required data to do the business logic (deciding something, inserting something, removing something), and the rest (HTTP controller, Kafka processor) will just validate and translate the incoming request and the outgoing response accordingly

The business logic (aka domain services and models) is just meant to “decide things”, in the sense of how should we modify the application state and/or what actions should be performed. But it doesn’t actually change the true application state, that will be done by the “infrastructure” and orchestrated by the “application service”. Fetching the data also isn’t done by the business logic, but what to fetch is decided by it. This can be either static, the input to the business logic tells you what it needs. Or dynamic, it could return a request for more data, similar to how it returns the change events and error events.

People sometimes describe the business logic as the invariants and rules around state modification and action taking.

It’s almost like the controller asks the business logic, hey, am I allowed to do this? Or, if I’ve got this and was told to do that, what should the result be?

So the business logic responds with things like: “You can’t do that”, “you should insert the following new user”, “you need to get me the users roles as well”.

The goal is to move as much as is possible into the pure side-effect free part of the code base, which is the business logic. But it’s not possible to move everything. If the requirement for the app is to send emails, you’ve got to send emails, no way to make that pure and side effect free. All useful applications will do side-effect and modify state. Sometimes you’ve got an app that only returns the input transformed, like say jq, but even that does a “print result back to user” at the end, which is an impure action.

But at least we’ve got as much as we can in a place that is easily testable with simple unit tests, and well isolated from other application complexities like I/O.

@depdev

Maybe to go further, I can mock all the components of the controller and the test cases might be easier to write (to isolate, just to test the if-elses, error handlings, etc within the controller code, without having to construct the needed values for the biz logic/view/db). I’ll definitely try this out.

Yes. Ideally you’re in a place where the controller only orchestrate. It actually doesn’t do I/O and doesn’t do any business logic. It just decides what/when to call from the infrastructure (the DB for example), and what/when to call from the business logic, and in what order and if some data returned by one must be piped to the other, etc. Meaning you can mock the infrastructure (like the DB) and the business logic, and just have a unit test that it’s doing the right orchestration.

Then the business logic you can also unit test, given some input does it return the output you’d expect (the right error/change/data-request events).

Finally you’re left with the “infrastructure”. This is the hardest bit, it’s what actually does I/O. But it also will interpret the events returned by the business logic, and convert them to the necessary I/O instructions. For example, it will take [userInsertedEvent, user] and it’ll have to convert that to the SQL to run on the DB.

Some people decide to write it so you can unit test this part on its own, it takes a domain event and returns the SQL for example.

And then there’s another part that actually runs the SQL, and that you don’t unit test, but integ test.

Testing the infrastructure is hard though either way. Some people go straight to integ tests, or others swap for some in-memory stub DB or something like that, but not all DB offer a API compatible in-memory replica.

Other things the “infrastructure” can do is perform actions. Call another service, scrape the web for something, send an email, publish a RabbitMQ message or Kafka message, etc. Those are all also trickier to test. But at least you’ve got everything else properly tested.

Finally, the infrastructure also handles rendering things back to the user or the screen. The “view” is actually part of your business logic, it returns what to return, like the HTML, but the infrastructure will actually render it to screen, HTML is just a data description of something to render, so marshalling the HTML back into an HTTP response to the browser and the browser rendering it to the user screen is all part of your application “infrastructure”.

@depdev

is it natural for the application service to grow more than the rest of the codebase, in terms of:

• having to handle errors
• if-elses to handle different outcomes/events
• initializing db/cache/api clients
• usage of test doubles (db stubs/mocks)
  and for most cases, the application service is the place to check out first in order to understand the general flow of the application?

It really depends. The closer to CRUD your application is, than yes. But the further away, the more will be in the change/view business logic part. Think a CPU/GPU heavy application versus a I/O heavy one. If your use case is more about just doing I/O, and routing request to the appropriate I/O operation, ya, the app-service/infra will be a big part of that. But if your use-case is to compute complicated things, or has a lot of rules/invariants over CRUD in terms of what is allowed to be changed and how, etc., more will be in the domain layer.

That said, you should try to think a bit a about infra/application-service as seperate.

For small apps, it’s good to make them the same. But orchestrating the steps between infra/domain on it’s own is a responsibility pretty different from calling a database or returning an HTTP response.

This is when sometimes separating that as well makes sense.

Generally MVC frameworks already do this. By the time your controller is called, the HTTP request was already processed, unmarshalled, authenticated, routed, and it’s payload converted to only a command + parameters. And when you returned from your controller, you’re not like sending packets back to the TCP connection, you’re returning the command result and the MVC framework will marshall that back and be responsible for sending it back as a HTTP response with all the headers and all into the TCP connection.

But the DB or calling other services, or sending emails/messages and all that isn’t always done by the framework. So you could choose to do that in the controller directly (small apps it’s okay), or actually create infrastructure components around those, the the controller just uses, similar to how it uses the business logic, and that makes the controller’s job purely to orchestrate.

[didibus]

I know my previous answer is already long and detailed and has a lot to digest. But I also wanted to add one more thing.

There’s two “famous” approach to making the business logic pure.

1. Return events that dictate what side-effect to perform. It’s then the application service that orchestrate the interleaving of business logic and side-effects as needed.

2. Inject the side-effect performing actions into the business logic. The business logic therefore calls the injected function/object to perform the side-effect where it’s needed, which is pseudo-pure, because in unit tests you can inject mocked and therefore pure functions/objects, but at runtime you can inject impure functions/objects that actually do side-effects. The business logic can then also orchestrate the interleaving.

Some people prefer #1, some prefer #2. Some will do a hybrid.

You can experiment with what your prefer. I prefer #1 generally. I like having the step-by-step orchestration laid bare at the top and limit the call stack depth. But others find it too annoying to structure things like that and prefer the more intuitive like “call the side-effect where in the chain it logically is needed but do it where you can easily inject a pure variant making the whole thing pure for tests”.

[Linus_Ericsson]

Regarding transaction instructions: yes, it is a data structure that contains commands whose resulting side-effects should be commited in some atomic manner.

This is of course could be similar to what a s-expression in clojure or other lisp would be, but with some extra logic to make sure it is atomic. However, I think the interesting part here is that you can compile our complicated business logic to something which runs much faster in a transaction (in Datomic there are some primitivies like add, retract, cas, but also, if needed, functions running in the transactor can be added), but all of these are communicating what to write with simple primitives for add, retract etc.