Now, I would like to add an arbitrary number of filters and/or transformations where there are the ... by starting from a configuration data structure. What I have up until now is:
But this isn’t working. I’m aware of the fact that my problem is that I’m trying to evalue arguments at compile time, but I can’t seem to find a working solution
In my opinion, the ->> macro is mainly a syntactic facility to reduce the syntactic clutter when you write code to pass some value through many functions. But for your use case where you have to build a pipeline based on some data specification, macros are not a very good choice. Clojure is a functional language, and functions can be used in many different configurations to achieve what you want. In your case, I believe transducers may be a good idea. Here is a draft to get you started:
I replaced your macro by add-steps by the function do-steps, removed the do-all in parse-steps as well as the backquotes on (filter, so that parse-steps will return a sequence of filtering transducers. But I haven’t tested it.
Hey thanks for the answer! Anyway I’m aware of transducers and I’m currently using them, that was my plan B
The point was to learn more about macros and since I’m working with largish data I’m currently using tesser to parallelize computations, so the code here is just an example what I’m really doing is (tesser/filter #(= (~@(keys d) %) ~@(vals d))) etc.
But I can always try to parallelize the transducer with core.async, though I’m not an expert with it