When does data-orientation get too much?

[orestis]

So recently at work I was trying to come up with a generic way to wrap some data-altering operations, to do stuff like capturing previous versions, doing denormalized updates and dispatching notifications.

For compatibility with a legacy system, I have constraints that dictate the format of the operations.

It’s an interesting topic, because you somehow need to:

A) keep previous results around to reuse in a later stage (eg save a record id to put in the notification queue)
B) keep the result of the “main” operation around to use for the notifications, and the return value
C) Have good error handling
D) keep a transaction open for some of those operations, but not for others

I started modeling operations with maps, did a dispatch function to execute them, tried to give them intent-revealing names, add a way to have named arguments and results to be reused etc etc.

Everything works fine and ticks off all the boxes - but then I realized I’ve implemented my own little mini-language and interpreter.

I understand the value of representing all those side effects as data, and it definitely opens up possibilities for the future (doing this in parallel, retrying operations, switching implementations around)… but it does seem a little silly.

I’m essentially doing all this to avoid a couple of nested let-bindings. So that’s why I’m asking, can data-orientation go too far?

[ikitommi]

Yes, it can, like anything one can over-engineer.

Have seen way too complex side effect models in Java, Scala and Clojure. In Java, tons of classes, injections and boilerplate, In Scala using the not-so-easy Free Monad and in Clojure, the data-driven incomplete mini-flow languages. I think there is room for making solid libraries in this space for Clojure, not sure if I have seen one yet. Libs like sci could help to build the flows using normal clojure code, as data.

Just today we are having a internal coffee & tech discussion about merging our pure and impure (imperative side effects) state machines into one. Both useful, but neither complete.

Few pointers:

• https://github.com/seancorfield/engine
• https://github.com/Day8/re-frame-async-flow-fx

[orestis]

I also saw GitHub - schmee/daguerreo: A DAG task execution engine for Clojure pop up recently, just as I finished my attempt. It does look designed for this use case:

Daguerreo is a library to create workflows using tasks. It takes care of dependency resolution, parallellism, retries and timeouts and let’s you focus on your business logic.

Which does seem something that normal Clojure evaluation will have a hard time dealing with while maintaining some readability.