# X\> & x\>\>: auto-transducifying thread macros (now with parallelizing |\>\> and =\>\>)

**URL:** https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122
**Category:** Showcase
**Tags:** macros, transducers
**Created:** [September 1, 2021, 6:03pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122 "2021-09-01T18:03:57Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![John\_Newman](https://clojureverse.org/user_avatar/clojureverse.org/john_newman/32/1254_2.png) [@John\_Newman](https://clojureverse.org/u/John_Newman)
#### Post date: [September 1, 2021, 6:03pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/1 "2021-09-01T18:03:57Z")

</div>

Hey folks, I just whipped up a little auto-transducifying thread macro last night. It basically searches for contiguous groups of transducers and comps them together into a function that sequences the values flowing in the thread through the transducers. First the code, then we’ll discuss:

```auto
(def transducables
  #{(symbol 'map)
    (symbol 'cat) 
    (symbol 'mapcat) 
    (symbol 'filter)
    (symbol 'remove) 
    (symbol 'take) 
    (symbol 'take-while) 
    (symbol 'take-nth) 
    (symbol 'drop) 
    (symbol 'drop-while) 
    (symbol 'replace) 
    (symbol 'partition-by) 
    (symbol 'partition-all) 
    (symbol 'keep) 
    (symbol 'keep-indexed) 
    (symbol 'map-indexed) 
    (symbol 'distinct) 
    (symbol 'interpose) 
    (symbol 'dedupe) 
    (symbol 'random-sample)})

(defn transducable? [form]
  (when (sequential? form)
    (contains? transducables (first form))))

(defn compose-transducer-group [xfs]
  (->> xfs
       (map #(apply (first %) (rest %)))
       (apply comp)))

(defn xfn [xf-group]
  (fn [args]
    (sequence
     (compose-transducer-group xf-group)
     args)))

(defmacro x>>
  "Just like ->> but first composes consecutive transducing fns into a function
   that sequences the last arguement through the transformers. Also, calls nth
   for ints. So:
   
   (x>> [1 2 3] 
        (map inc) 
        (map (partial + 2)))
   Becomes:
   
   ((xfn [[map inc] 
          [map (partial + 2)]]) 
    [1 2 3])"
  [x & threads]
  (let [forms (->> threads
                   (partition-by transducable?)
                   (mapv #(if-not (and (transducable? (first %))
                                       (second %))
                              %
                              (list (list `(xfn ~(mapv vec %))))))
                   (apply concat))]
    (loop [x x, forms forms]
      (if forms
        (let [form (first forms)
              threaded (cond (seq? form)
                             (with-meta `(~(first form) ~@(next form) ~x) (meta form))
                             (int? form)
                             (list `nth x form)
                             :else
                             (list form x))]
          (recur threaded (next forms)))
        x))))

(defmacro x>
  "Just like -> but first composes consecutive transducing fns into a function
   that sequences the second arguement through the transformers. Also, calls nth
   for ints. So:
   
   (x> [1 2 3]
       (conj 4)
       (map inc)
       (map (partial + 2))
       2)
   Becomes like:
   
   (nth
    ((xfn [[map inc] [map (partial + 2)]]) 
     (conj [1 2 3] 
           4)) 
    2)"
  [x & threads]
  (let [forms (->> threads
                   (partition-by transducable?)
                   (mapv #(if-not (and (transducable? (first %))
                                       (second %))
                            %
                            (list (list `(xfn ~(mapv vec %))))))
                   (apply concat))]
    (loop [x x, forms forms]
      (if forms
        (let [form (first forms)
              threaded (cond (seq? form)
                             (with-meta `(~(first form) ~x ~@(next form)) (meta form))
                             (int? form)
                             (list `nth x form)
                             :else
                             (list form x))]
          (recur threaded (next forms)))
        x))))

```

Neat, right? Pretty simple. Why would you want to do this? Well, for one thing:

```auto
(->> (range 10000000)
     (map inc)
     (filter odd?)
     (mapcat #(do [% (dec %)]))
     (partition-by #(= 0 (mod % 5)))
     (map (partial apply +))
    ;; (mapv dec)
     (map (partial + 10))
     (map #(do {:temp-value %}))
     (map :temp-value)
     (filter even?)
     (apply +)
     time)

```

Returns:

```auto
"Elapsed time: 5768.05707 msecs"
5000054999994

```

Whereas:

```auto
(x>> (range 10000000)
     (map inc)
     (filter odd?)
     (mapcat #(do [% (dec %)]))
     (partition-by #(= 0 (mod % 5)))
     (map (partial apply +))
    ;; (mapv dec)
     (map (partial + 10))
     (map #(do {:temp-value %}))
     (map :temp-value)
     (filter even?)
     (apply +)
     time)

```

Returns:

```auto
"Elapsed time: 2102.793256 msecs"
5000054999994

```

Twice the speed with basically the same code.

The more transducers you can get lined up contiguously, the less boxing you’ll have in your thread. Let’s uncomment the `(mapv dec)` that is present in both the threads above. Because `mapv` is not a transducer, items get boxed halfway through our thread. As a result our performance degrades slightly for `x>>`.

First `->>`:

```auto
"Elapsed time: 4590.188052 msecs"
44999977000016

```

Hmm, `->>` actually goes faster now, perhaps due to `mapv` removing some laziness.

But for `x>>`:

```auto
"Elapsed time: 2351.326542 msecs"
44999977000016

```

So we lost some speed due to the boxing, but we’re still doing way better than the default thread macro. Point is, if you want to maximize performance, try to align your transducers contiguously.

One more perc with `x>` and `x>>`: a naked integer in a thread becomes an `nth` on the value threading through, so you can use them as replacements for `get-in` for most cases with heterogeneous nestings:

```auto
(let [m {:a {:b [0 1 {:c :res}]}}]
  (x> m :a :b 2 :c))

```

That’s just a personal preference but some might not want the change in semantics. Also, `x>` is different in that if a transducer is passed in, it acts like thread-last. This too is a semantic divergence from `->` but in my experience we rarely ever use functions that take a function as a first argument in thread-first threads, so there probably won’t be too much confusion - and this makes `x>` more useful.

Next steps: integrate Christoph Grand’s xforms transducers (maybe automatically, anaphorically, for the core fns, like `reduce` and `into`?)

Anyway, I just whipped this up, so there may be downsides to this formalism I haven’t noticed, but that’s what this post is for - to debate the merits or lackthereof! 🙂 If it’s not a bad idea, I’ll probably wrap it up in a lib for general usage - but you are hereby granted, by the power invested in me, the inalienable right to copy and paste any of the words or code in this post anywhere you want. 😉 (edit: oh, and the meat of the threading code is taken from clojure.core anyway) Any other features or constraints y’all’d like to see in this code pattern?

Happy hacking!

(edit: After some discussion below, I released the library version of this idea here: [johnmn3/injest](https://github.com/johnmn3/injest))

---

<div class="post-metadata">

### Author: ![bsless](https://clojureverse.org/letter_avatar_proxy/v4/letter/b/6f9a4e/32.png) [@bsless](https://clojureverse.org/u/bsless)
#### Post date: [September 1, 2021, 6:54pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/2 "2021-09-01T18:54:49Z")

</div>

Combine it with a slightly faster comp 🙂

> <https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L156>

Edit: Why `(symbol 'foo)` and not just `'foo`?

---

<div class="post-metadata">

### Author: ![John\_Newman](https://clojureverse.org/user_avatar/clojureverse.org/john_newman/32/1254_2.png) [@John\_Newman](https://clojureverse.org/u/John_Newman)
#### Post date: [September 1, 2021, 7:12pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/3 "2021-09-01T19:12:41Z")

</div>

Nice! I thought `'foo` might expand to `(quote foo)` and maybe not match by equality, but maybe not. I’ll bang on it and see, but it’s a compile time thing anyway, so shouldn’t affect the run time.

---

<div class="post-metadata">

### Author: ![joinr](https://clojureverse.org/user_avatar/clojureverse.org/joinr/32/1861_2.png) [@joinr](https://clojureverse.org/u/joinr)
#### Post date: [September 1, 2021, 9:11pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/4 "2021-09-01T21:11:09Z")

</div>

fusing contiguous map operations into a single function, ditching partial might be useful (takes like maybe 150 ms off on my end).

```clojure
 [map (fn [arg] (as-> arg x
                      (apply + x)
                      (+ 10 x)
                      {:temp-value x}
                      (x :temp-value)))]

```

Curious if more substantive fusing could be had, along with a better comp implementation like ben mentioned. Kind of opens the door toward a little analyzer and maybe more optimizations. Cool idea.

---

<div class="post-metadata">

### Author: ![John\_Newman](https://clojureverse.org/user_avatar/clojureverse.org/john_newman/32/1254_2.png) [@John\_Newman](https://clojureverse.org/u/John_Newman)
#### Post date: [September 1, 2021, 9:21pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/5 "2021-09-01T21:21:48Z")

</div>

True. Granted, the author probably _should_ do that fusing themselves, in any threading scenarios, as a matter of refactoring. But sure, why not optimize all the things? Especially if the side-effect semantics remain the same. Hadn’t considered the map fusing idea - I’ll stew on it, thx!

---

<div class="post-metadata">

### Author: ![John\_Newman](https://clojureverse.org/user_avatar/clojureverse.org/john_newman/32/1254_2.png) [@John\_Newman](https://clojureverse.org/u/John_Newman)
#### Post date: [September 2, 2021, 2:18am UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/6 "2021-09-02T02:18:56Z")

</div>

In related news: [https://www.discovermagazine.com/mind/your-brain-is-not-a-computer-it-is-a-transducer](https://www.discovermagazine.com/mind/your-brain-is-not-a-computer-it-is-a-transducer) lol

---

<div class="post-metadata">

### Author: ![bsless](https://clojureverse.org/letter_avatar_proxy/v4/letter/b/6f9a4e/32.png) [@bsless](https://clojureverse.org/u/bsless)
#### Post date: [September 2, 2021, 4:51am UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/7 "2021-09-02T04:51:25Z")

</div>

The problem with all of this is that we’re turning ourselves into mini compilers, implementing heuristics we have empirically learned to transform our code.  
What we’re really looking for is a compiler which can do inlining, abstraction, constant propagation and constant folding.  
You could probably do something like that with core.logic, if you had enough time and funding. Looks like a good subject for a thesis or three.

---

<div class="post-metadata">

### Author: ![John\_Newman](https://clojureverse.org/user_avatar/clojureverse.org/john_newman/32/1254_2.png) [@John\_Newman](https://clojureverse.org/u/John_Newman)
#### Post date: [September 2, 2021, 3:31pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/8 "2021-09-02T15:31:01Z")

</div>

The problem with human refactoring? Or do you mean the problem of trying to reason through these optimizations?

Well this particular piece of fruit was hanging lower than I’d realized. I’d been wanting to make it for a while but thought it would be way hairier. I’ve been working on an open source front end lib for higher order components, based on higher order functions, which can take transducers in various places, since it kinda fit as a “transducing context” and I sorta stumbled on the realization that this particular optimization could be simpler than I’d previously thought.

I guess another low hanging piece: If we lifted out [cljr-thread-last-all](https://github.com/clojure-emacs/clj-refactor.el/wiki/cljr-thread-last-all) from clj-refactor.el, then we could wind vanilla code into a thread-last and apply this auto-optimization to vanilla code as well 🤔 Essentially a code-walker than can auto-wind up any nested calls into a thread-last form and then apply this macro to fuse up the transducers. I’d probably add that to [perc](https://github.com/johnmn3/perc), so any forms within a perc fn get auto-optimized.

The map fusion may be low hanging as well, with simple code walking. An actual compiler that carries an environment around through the code… Maybe ClojuristsTogether could fund something like that one day. I’d throw down on it.

---

<div class="post-metadata">

### Author: ![joinr](https://clojureverse.org/user_avatar/clojureverse.org/joinr/32/1861_2.png) [@joinr](https://clojureverse.org/u/joinr)
#### Post date: [September 2, 2021, 10:02pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/9 "2021-09-02T22:02:18Z")

</div>

hotspot gets a (big) vote too. the question then becomes whether the fused/optimized code actually improves upon what hotspot is able to detect e.g. via `comp` and apply its optimizations to. I think in the case of the original macros here, the benefit is largely just moving away from seq generation and going into loops and composed functions.

I am uncertain (testing would be interesting), since hotspot apparently likes small methods instead of big ones, so it can more aggressively inline. Maybe fusion isn’t all that great and `comp` already plays to the jit’s strengths. Or maybe not.

In the broader context of optimizing “mini compilers” I would certainly welcome that over the burden of trying to write a general purpose optimizing compiler. E.g. optimize a pipeline into something 2x or more performant seems plenty fine. We already do that idiomatically in clojure anyway with certain abstractions (arraymap vs. persistentmap; chunked seqs; etc.). Why not have drop-in forms that can try to speed up the local context using limited hueristics and/or passes?

Note: a lot of this could be done on top of tools.analyzer (e.g. more agressive optimizations) and/or core.typed even. I definitely think it would be a research project, but perhaps well scoped. Again, the benefits are uncertain (absent extreme fusion and automatic lifting e.g. of primitive operations…something like turn your -\>\> pipeline into an optimized primitive-friendly bytecode representation amenable to type hinting).

---

<div class="post-metadata">

### Author: ![John\_Newman](https://clojureverse.org/user_avatar/clojureverse.org/john_newman/32/1254_2.png) [@John\_Newman](https://clojureverse.org/u/John_Newman)
#### Post date: [September 2, 2021, 11:48pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/10 "2021-09-02T23:48:31Z")

</div>

Yeah, it should be noted as well:

1. Because transducers have different laziness semantics, you can’t just auto-optimize _all_ vanilla code naively without some stuff blowing up, so it’s best that the caller is aware of the change of semantics at the call site, via a known macro or something
2. Some stateful transducers may be optimized for single-threaded performance and may not produce expected results in some multi-threaded scenarios… not sure if that applies to the macros above, as some context outside the thread would be orchestrating mutli-threading, which I don’t think would usually bang on those transducer’s internal state independently… But removing the stateful transducers from the above macros may lend the formalism to integrating more parallelism into the threading semantics, with maybe like a `px>>` that auto-parallelizes the stateless `map`ping transducer.

So small macros like this that provide explicit performance improvements (with associated caveats) may be ideal for exploring the space of possible future optimizations. There are so many different avenues of optimization and some of them are not friendly to one another, so empowering the caller to choose the correct optimization strategy for their needs requires less forethought from the tool developers. The problem is providing powerful tools that fit with developers existing UX ergonomics - these thread macros are a perfect fit for auto-transducification, where it already fits with developers existing coding ergonomics.

So I agree, and I’ll probably forgo the map-fusion optimization, to try to deliver an optimization semantic that is more easily understood by the caller: this optimization `x>>` is about fusing transducers in a thread. `px>>` only works for stateless transducers, etc. (though it’d be interesting to explore if map fusion pays more dividends in multi-threaded `map`ping-transduction scenarios)

An actual optimizing compiler is outside of the scope of this idea, but this idea could def be used somewhere in an optimizing compiler, if the caller is aware of the changed laziness semantics, or if the compiler can know the call won’t blow up memory.

What about the two other semantic changes I made: `nth` on integers and allowing the thread-last semantic on transducers in a thread-first `x>` form?

I’d argue both are ergonomically additive changes, as, for integers, a naked int would have always thrown an exception anyway, and, for `x>`, without allowing for the apparent thread-last semantic on transducers, there’d be no point in having a thread-first version anyway. And _very_ rarely do we ever thread-first some produced function into the first param position of some collection operator in a thread-first macro, so callers will likely know, “Okay, this is a transducer-thread-first macro and the function at this point in the thread is a function that usually takes the collection last, so this is probably the transducer version, so I can imagine this as being like a thread-last semantic at this point in the thread.” And no other semantic could be expected for a `x>`-like form, because transducers are defined according to their step function, not their collection params. So it’s either that weirdness or no `x>` macro at all, and I figure it’s better off having it than not, so you can grab for those thread-first scenarios.

Anyway, if there’s no objections to those two semantic differences, I’ll ship them as is.

---

<div class="post-metadata">

### Author: ![bsless](https://clojureverse.org/letter_avatar_proxy/v4/letter/b/6f9a4e/32.png) [@bsless](https://clojureverse.org/u/bsless)
#### Post date: [September 3, 2021, 6:27am UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/11 "2021-09-03T06:27:21Z")

</div>

I’ll give a simple example of what I mean  
Let’s say you write

```clojure
(get-in m [:a :b])

```

Since you know the keys at compile time, you could substitute with:

```clojure
(-> m (get :a) (get :b))

```

which is faster  
But it’s based on human knowledge. A compiler would have done something different:

```clojure
(get-in m [:a :b])
((fn [m ks] (reduce get m ks)) m [:a :b])
(reduce get m [:a :b])
(reduce (get m :a) [:a])
(reduce (get (get m :a) :b) [])
(get (get m :a) :b)

```

Inline and partially apply as much as possible, without apriori knowledge

---

<div class="post-metadata">

### Author: ![John\_Newman](https://clojureverse.org/user_avatar/clojureverse.org/john_newman/32/1254_2.png) [@John\_Newman](https://clojureverse.org/u/John_Newman)
#### Post date: [September 3, 2021, 10:11pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/12 "2021-09-03T22:11:35Z")

</div>

There’s now a repo up on Github where you can grab it using deps.edn. Consider it alpha for now. Once it has some usage and any issues are ironed out, I’ll push up an artifact to clojars.

(edit: oh, and there’s also a new extension feature `reg-xf` that allows the user to extend the macros by adding extra transducers that they’re able to recognize. An example of bringing in `cgrand/xforms` is in the readme)

> **[GitHub - johnmn3/injest: x\> & x\>\>: Auto-transducifying thread...](https://github.com/johnmn3/injest)**
>
> x\> & x\>\>: Auto-transducifying thread macros. Contribute to johnmn3/injest development by creating an account on GitHub.

---

<div class="post-metadata">

### Author: ![DrLjotsson](https://clojureverse.org/user_avatar/clojureverse.org/drljotsson/32/2602_2.png) [@DrLjotsson](https://clojureverse.org/u/DrLjotsson)
#### Post date: [September 4, 2021, 2:59pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/13 "2021-09-04T14:59:55Z")

</div>

Cool! I often end my `->>` with `(into [])` or `(into {})`. Would it make sense to detect if a group of transducables is followed by an `into` and use that as the transducing function instead of `sequence`?

---

<div class="post-metadata">

### Author: ![John\_Newman](https://clojureverse.org/user_avatar/clojureverse.org/john_newman/32/1254_2.png) [@John\_Newman](https://clojureverse.org/u/John_Newman)
#### Post date: [September 4, 2021, 4:39pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/14 "2021-09-04T16:39:08Z")

</div>

Not a bad idea, but honestly it’d probably only add a few percent increase, based on my testing. Also, using `sequence` allows us to know we’re getting laziness semantics a little bit closer to the laziness of the pipeline functions we’re replacing with transducers. And you can grab for transducers that are potentially more lazy, like, you could pass in cgrand’s `x/into` and follow it with a `(take 5)` and you’ll know the `x/into` _can_ short-circuit after the first chunk, not having to realize the whole sequence (not tested yet).

In general, I’m pretty amazed at the performance of `sequence` compared to `transduce` (which `into` uses under the hood) given the extra caching and laziness semantics you get out of it - just from my minimal testing. But it would be useful to build a performance test suite to compare all these different scenarios. If transducing groups that end in an `into` (or any other transducer consumer) show significant dividends in various “general” situations, rather than `sequence`ing them, then I’d try it.

FYI, the ClojureScript version is currently broken. Looks like I’m going to have to drop into the cljs.analyzer.api to resolve names 😕 Should be fixed up this weekend though.

---

<div class="post-metadata">

### Author: ![John\_Newman](https://clojureverse.org/user_avatar/clojureverse.org/john_newman/32/1254_2.png) [@John\_Newman](https://clojureverse.org/u/John_Newman)
#### Post date: [September 4, 2021, 10:41pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/15 "2021-09-04T22:41:42Z")

</div>

Got a workaround for ClojureScript, listed in the readme. Spin up a repl with criterium and let me know if you find anything odd:

```clojure
clj -Sdeps \
    '{:deps 
      {io.github.johnmn3/injest {:git/tag "v0.1-alpha.6" :git/sha "0a5f2df"}
       criterium/criterium {:mvn/version "0.4.6"}
       net.cgrand/xforms {:mvn/version "0.19.2"}}}'

```

---

<div class="post-metadata">

### Author: ![John\_Newman](https://clojureverse.org/user_avatar/clojureverse.org/john_newman/32/1254_2.png) [@John\_Newman](https://clojureverse.org/u/John_Newman)
#### Post date: [September 8, 2021, 4:33am UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/16 "2021-09-08T04:33:07Z")

</div>

Quick update on the API that is settling in and some extra commentary:

#### 1. Extending `injest` with `reg-xf!`

You can now register one or more transducers like this `(injest/reg-xf! x/reduce)`. Works in ClojureScript too, but you have to register the names from a Clojure namespace for now.

#### 2. As a replacement for `get-in`

We’re now wrapping both integers and strings in a `get`, so that you can concisely thread into basically anything:

```clojure
(let [m {1 {"b" [0 1 {:c :res}]}}]
  (x> m 1 "b" 2 :c name)) ;=> "res"

```

I can’t tell you how many times I’ve been using thread macros and I’ve wanted this feature. Why not make numbers and strings usable for lookups at those locations? It just makes sense.

On the subject of ClojureScript, for the above example threads, I’m seeing a _6 times_ (or more) improvement in speed! This is some _very_ low hanging fruit, folks!

With an initial value of `(range 1000000)`, ClojureScript’s `->>` produces:

```clojure
"Elapsed time: 3523.186678 msecs"
50005499994

```

While the `x>>` version produces:

```clojure
"Elapsed time: 574.145888 msecs"
50005499994

```

That’s free money, y’all!

I guess the question is, why wouldn’t you just default to `x>` and `x>>` always, as a matter of convenience? You get better path navigation through data structures and even if you’re not using a transducer in it now, you might want to grow it that way in the future - so why not just opt for the transducing version now?

And your junior Clojure devs don’t even have to understand transducers. At least not initially. They just have to learn the semantics of the thread-last macro, then give them the `x>>` macro and then let them go to town. It’s like training-wheels for transducers 🙂

In the rare case you really need to manage things in a threading pipeline as a lazily computed, infinite sequence, then you can just go back to `->>`, no big deal, but that’s really the edge case.

---

<div class="post-metadata">

### Author: ![bsless](https://clojureverse.org/letter_avatar_proxy/v4/letter/b/6f9a4e/32.png) [@bsless](https://clojureverse.org/u/bsless)
#### Post date: [September 8, 2021, 6:04am UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/17 "2021-09-08T06:04:20Z")

</div>

Are you familiar with [clj-fast](https://github.com/bsless/clj-fast)?  
It has a lot in common.  
There is a balance to be struck, though, with code-rewriting macros, especially when they change the actual code being executed. I still need to clarify that in the documentation

---

<div class="post-metadata">

### Author: ![John\_Newman](https://clojureverse.org/user_avatar/clojureverse.org/john_newman/32/1254_2.png) [@John\_Newman](https://clojureverse.org/u/John_Newman)
#### Post date: [September 8, 2021, 1:43pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/18 "2021-09-08T13:43:15Z")

</div>

Aye, I’ve checked it out! Pretty awesome stuff. I’ll probably mention your work (and @joinr’s work) in a references section on the repo, as I build it out.

I think you do mention it somewhere in the docs - macro’ized versions of functions have different ergonomics, can’t be passed around etc. Fortunately, `->>` was born with that limitation, so those ergonomics remain consistent for `x>>`. The practical difference here pertains to eagerness on intermediate values, but I’m betting user’s won’t need to even be aware of that difference, from a UX perspective, for 99% of their code.

There’s still some area for improvement. I’d like to replace the `get` with a `get-or-nth-for-lists` so that the path navigation works for numbers on lists as well, but I’m reluctant to introduce a new runtime function in such a thin, abstract macro. I’d only add that if I can ensure it doesn’t introduce a significant performance hit vs using the plain `get` at runtime. Ideally, it’d outperform `get`. Might explore the strategy in `clj-fast.core/val-at` for that.

Could also wrap the thread-top-level keywords in a `get` form, but I recently read somewhere that `(:x m)` is faster than `(get m :x)`, so I left them as is - worth investigation though.

Ultimately, though, I view the purpose of this macro as an ergonomic convenience over the already existing performance tools in Clojure (transducers), moreso than inventing any new performance features. It just tries to help you introduce transducers to your codebase while maintaining your other existing expectations as much as possible.

---

<div class="post-metadata">

### Author: ![bsless](https://clojureverse.org/letter_avatar_proxy/v4/letter/b/6f9a4e/32.png) [@bsless](https://clojureverse.org/u/bsless)
#### Post date: [September 8, 2021, 4:20pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/19 "2021-09-08T16:20:46Z")

</div>

The shortest code path is actually `(m :x)`, including a nil-check.  
Regarding eagerness, looking at the final element is sufficient.  
If it is `(into coll ,,,)` you can turn it into transduce.  
If not, you can turn it into a `sequence`.

I’ve been considering writing a macro like that as well, so you’ll excuse me if I refrain from looking at your implementation until I do 🙂

---

<div class="post-metadata">

### Author: ![joinr](https://clojureverse.org/user_avatar/clojureverse.org/joinr/32/1861_2.png) [@joinr](https://clojureverse.org/u/joinr)
#### Post date: [September 8, 2021, 4:36pm UTC](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122/20 "2021-09-08T16:36:00Z")

</div>

> We’re now wrapping both integers and strings in a `get`, so that you can concisely thread into basically anything

It’s your macro, so you extend the language as you like. However, this is notably differing from clojure semantics, e.g. numbers, strings, etc. are not applicable. Maybe there is a reason for this at the language design level (buried in google groups discussions from a decade ago). I think the reason for strings that they leave a broad ambiguity: since strings are indexed, should they act as associative containers like vectors, mapping int → val, or are they implicit keys into an associative container? Numbers have a similar ambiguity, since there are slightly different paths between get/nth depending on support for Indexed. The only types that are explicitly viewed as keys are keywords and symbols, given their IFn implementation.

Since you mentioned targeting newcomers, it probably bears mentioning that the semantics of the codegen in this macro differ from clojure.core since in your (now technically DSL), numbers and strings have implicit IFn implementations with a specific interpretation (where, in the case of strings, they could have multiple interpretations). In the language of the progenitor, this is perhaps “complecting” things. In theory I don’t think it’s a big deal, but in practice it means that

> then you can just go back to `->>`

does not hold if the details of your DSL are relied on (e.g. strings/numbers interpreted specifically as keyword accessors). This could be jarring to consumers, specifically newcomers, when they are faced with the sudden, perhaps inscrutable error of java.lang.Long cannot be cast to clojure.lang.IFn or java.lang.String cannot be cast to clojure.lang.IFn .

[Next page](https://clojureverse.org/t/x-x-auto-transducifying-thread-macros-now-with-parallelizing-and/8122.md?page=2)
