Comprehensions are vile, or how I learned to love computational pipelines

[tyohDeveloper]

As you can tell from my subject line. I hate loop-like comprehensions. I’m looking for some help eliminating them. I just need a few simple macros.

Rather than

(partition 3 (map f (interleave x y z)) 

Why not

(kill-all-for-loop-recur [map f] & seqs)
(kill-all-for-loop-recur [filter f] & seqs)
(kill-all-for-loop-recur [fold f s] & seqs)

Naturally I’d Like to use #([%1 %2 %3]) to return the three input sequences.

Which comes up with the next challenge. I’d like default the partition into the same number of sequences. Or a different number of them.

Finally, change the response from row major back to column major. The last is a particular pain. It’s easy to stream pipelines in and out. Having to reform pipelines is messy.

Does anyone have a small macro library to help get rid of comprehensions?

[jjttjj]

Could you provide some example input/expected output data? I’m not entirely sure what you’re asking. But it seems like the answer might involve transducers. In particular I’m reminded of transjuxt from the xforms library but that might be overkill.

[vlaaad]

So you want to process 3 colls sequentially looking at corresponging elements? That looks just like a map — it accepts multiple collections.
Example:

(let [;; some colls
      ints [1 3 5]
      keys [:a :b :c]
      booleans [true true false]
      ;; mapping function `f`
      f str]
  (map #(map f %) (map vector ints keys booleans))) 
; => (("1" ":a" "true") ("3" ":b" "true") ("5" ":c" "false"))

[tyohDeveloper]

I think I’ll look at transducers. They may address some of the issues I am experiencing. I don’t have experience with them. I’ll look into them.

Those would be helpful for creating software busses. I don’t think they would help in general graphs. I’m considering trying to use Clara to automate some of the work. I think that would help a lot. The downside with expert systems is they can lead to the principle of maximum astonishment. Clojure doesn’t have great debugging tools.

Below I have a tiny example of the general concept. It won’t work great with large sequences. It couldn’t be scaled to a significant parallel system. Even a multi-threaded programming. I think it would be faster to handle each step in a single thread, accepting the blocking of long inline computations. The problem of infinite sequences aren’t really addressed at all. I don’t have a macro library to make it easier. That might help.

The real downside is it still needs an imperative style of programming. Not comprehensions, thank goodness. The programmer must still orchestrate all of the map/reduce/filter kind of behaviors. A declarative library that can be mixed and matched is really what I’m looking for.

This is a trivial example of the concept in general. There is only one transform step. If the problem is more complex, it’s easy enough to orchestrate parts. But the orchestration doesn’t create lego like parts. A simple software bus could give one something closer. Anyway, this is my starting point.

(defn lump [& colls]
  "Column major to row major"
   (partition
   (count colls)
   (apply interleave colls)))

(defn unlump [coll]
  "Row major to column major"
  (map
   #(map
     (fn [i] (nth coll i) %)
     (range (count (first coll))))
   coll))

(defn transform [coll]
  "Trivial transformation, select first two columns."
  (map #(take 2 %1) coll))

;; Example
(def abc [10 11 12])
(def xyz [20 21 22])

(def abc-xyz (lump abc xyz (range)))
(def lazy-abc-xyz (realized? abc-xyz))

(def result (unlump (transform abc-xyz)))
(def lazy-result (realized? result))

(def first-result (nth result 0))
(def lazy-first-result (realized? first-result))

(def second-result (nth result 1))
(def lazy-second-result (realized? second-result))

(println "lumped" abc-xyz "realized?" lazy-abc-xyz)
(println "transform [[this is a trival example]" result "realized?" lazy-result)
(println "first result" first-result "realized?" lazy-first-result)
(println "second result" second-result "realized?" lazy-second-result)

[jiyinyiyong]

tip: be careful about your Markdown syntax https://guides.github.com/features/mastering-markdown/#GitHub-flavored-markdown .

[tyohDeveloper]

jiyinyiyong Did I get something wrong? The only markup I did was to indent the code four spaces. Should I have done more?

[didibus]

I’m not sure I’m able to understand. What shape of data do you have, and what transformations are you looking to make on it?

[teodorlu]

You might be able to format Clojure code specifically with something like

```clojure
(prn :hello)
````