I think I’ll look at transducers. They may address some of the issues I am experiencing. I don’t have experience with them. I’ll look into them.
Those would be helpful for creating software busses. I don’t think they would help in general graphs. I’m considering trying to use Clara to automate some of the work. I think that would help a lot. The downside with expert systems is they can lead to the principle of maximum astonishment. Clojure doesn’t have great debugging tools.
Below I have a tiny example of the general concept. It won’t work great with large sequences. It couldn’t be scaled to a significant parallel system. Even a multi-threaded programming. I think it would be faster to handle each step in a single thread, accepting the blocking of long inline computations. The problem of infinite sequences aren’t really addressed at all. I don’t have a macro library to make it easier. That might help.
The real downside is it still needs an imperative style of programming. Not comprehensions, thank goodness. The programmer must still orchestrate all of the map/reduce/filter kind of behaviors. A declarative library that can be mixed and matched is really what I’m looking for.
This is a trivial example of the concept in general. There is only one transform step. If the problem is more complex, it’s easy enough to orchestrate parts. But the orchestration doesn’t create lego like parts. A simple software bus could give one something closer. Anyway, this is my starting point.
(defn lump [& colls]
"Column major to row major"
(apply interleave colls)))
(defn unlump [coll]
"Row major to column major"
(fn [i] (nth coll i) %)
(range (count (first coll))))
(defn transform [coll]
"Trivial transformation, select first two columns."
(map #(take 2 %1) coll))
(def abc [10 11 12])
(def xyz [20 21 22])
(def abc-xyz (lump abc xyz (range)))
(def lazy-abc-xyz (realized? abc-xyz))
(def result (unlump (transform abc-xyz)))
(def lazy-result (realized? result))
(def first-result (nth result 0))
(def lazy-first-result (realized? first-result))
(def second-result (nth result 1))
(def lazy-second-result (realized? second-result))
(println "lumped" abc-xyz "realized?" lazy-abc-xyz)
(println "transform [[this is a trival example]" result "realized?" lazy-result)
(println "first result" first-result "realized?" lazy-first-result)
(println "second result" second-result "realized?" lazy-second-result)