Transducers sliding window, and unsliding window

Just to learn about transducers, and to get better at understanding Clojure, I did some exercises.
As an example, I wanted to make a pipeline that first multiplies each value by 10, and then takes the average with a running average of size 3.
turns out partition-all with 2 parameters does not make a transducer. So I found the sliding function online, which is a stateful transducer.
Then I could happily write

(sequence 
  (comp
    (map #(* 10 %))
    (sliding 3 1) 
    (map (fn [spls] (/ (apply + spls) 3)))) 
 (range 10))

This works nicely, and is very convenient. Also it is super powerful compared to imperative looping, because this could now be used on a stream of data etc.

So I wanted to generalize. I’m now able to create sub-structures inside the transducing loop. The remaining point is the inverse operation: to flatten sub-structures inside the transducing loop.
I’m having a hard time grasping how to do it properly.
I started out with a similar structure to the sliding function. But I’m totally in the dark on how to do this.
There is no way I have found to do this, except to put flatten outside the transducing part. Which in this case, is cheating.
Is it even possible?

Just to be clear, this is definitely outside my comfort zone in Clojure. Which is why I’m doing this, so I can learn.

(defn sliding "works as partition-all n m, but with transducers. (partiton-all n has transducer)
burgeport at https://clojure.atlassian.net/browse/CLJ-1858, but I'm not 100% positive if this is a bug...
taken from https://gist.github.com/nornagon/03b85fbc22b3613087f6
it is very similar to the partition-all source (https://github.com/clojure/clojure/blob/clojure-1.10.1/src/clj/clojure/core.clj#L7240)"
  ([^long n] (sliding n 1))
  ([^long n ^long step]
   (fn [rf]
     (let [a (java.util.ArrayDeque. n)]
       (fn
         ([] (rf))
         ([result]
          (let [result (if (.isEmpty a)
                         result
                         (let [v (vec (.toArray a))]
                           ;;clear first!
                           (.clear a)
                           (unreduced (rf result v))))]
            (rf result)))
         ([result input]
          (.add a input)
          (if (= n (.size a))
            (let [v (vec (.toArray a))]
              (dorun (take step (repeatedly #(.removeFirst a))))
              (rf result v))
            result)))))))

(defn unsliding
  ([^long n]
   (fn [rf]
     (let [a (java.util.ArrayDeque. n)]
       (fn
         ([] (rf))
         ([result]
          (do
            (println "r" result)
            (rf result)))
         ([result input]
          (do
            (if (seqable? input)
              (do (println "i" input)
                  ;(rf result input)
                  ;=> ([0 1 2 3] [1 2 3 4] [2 3 4 5] [3 4 5 6] [4 5 6 7] [5 6 7 8] [6 7 8 9] [7 8 9])
                  ;(rf (first input) (rest input))
                  ;=> ((1 2 3) (2 3 4) (3 4 5) (4 5 6) (5 6 7) (6 7 8) (7 8 9) (8 9))
                  (rf (rf result (first input)) (rest input))
                  ;=> (0 (1 2 3) 1 (2 3 4) 2 (3 4 5) 3 (4 5 6) 4 (5 6 7) 5 (6 7 8) 6 (7 8 9) 7 (8 9))
                  )
              (rf result input))))
         )))))

(comment
  (sequence (sliding 2) (range 10))
  ;=> ([0 1] [1 2] [2 3] [3 4] [4 5] [5 6] [6 7] [7 8] [8 9] [9])
  (sequence (comp (sliding 4) (unsliding 4)) (range 10))
  ; would like (0 1 2 3 1 2 3 4 2 3 4 5 ...) etc..
  )

Any input appreciated!

1 Like

Would you care to specify what this means? Function signature or an example, perhaps?

Edit: your trailing comment form clarifies this quite a bit.

Edit 2: I was thinking that (map flatten) might work, but it doesn’t. Good question. Hoping you’ll get some good responses.

Yes. that sentence got weird… I mean that the sliding fn makes lists that the next function receives as elements, conversely the unsliding fn would take the lists it receives and make the function after that receive single elements.
so (range 4), passed into (sliding 2 1) becomes '([0 1] [1 2] [2 3] [3]) and then, if fed to the theoretical unsliding fn, it would become '(0 1 1 2 2 3 3) Just like flatten BUT with the important difference of actually being a transducer, so it could be used in the same kind of pipeline

Calling the action the transducer does a “loop” might be just wrong. Sorry for the confusion. I’m still learning

1 Like

Couldn’t you flatten the incoming data inside the transducer and call the next function once for each element? No state needed inside the transducer.

I think you want the mapcat transducer.

2 Likes

This works nicely.

(sequence (comp (sliding 2 1) (mapcat identity)) (range 4)) 
;=> (0 1 1 2 2 3 3)

gives the expected output.
Thank you for your helpful answer.

I think you can skip mapcat identity and just cat:

sequence (comp (sliding 2 1) cat) (range 4))
;; (0 1 1 2 2 3 3)
1 Like

Yes. This also works. It is also more concise. As this whole thread clearly is a result of me not remembering these functions, I’ll have to remind myself to look through the clojure.core api more often.
Thank you.

1 Like