Looking for opinion about some macro I've been working on

Hey all,

After the whole Racket2 chat, which if you are not aware, includes a proposal to get rid of s-expressions in favor of a more familiar syntax to try and attract non Lisp devs to it, I started to toy with the idea of why is it people struggle with the Lisp syntax, and could a simple macro help make it more familiar.

With my first pass, I ended with a macro I called fam which allowed one to write code as such:

(fam
   var data = [{:id 1 :name "John"} {:id 1 :name "joHn"}]
   var group-by-fn = :id
   (group-by group-by-fn data))
;=> {1 [{:id 1, :name "John"} {:id 1, :name "joHn"}]}

(fam
    var x = 10
    var y = 30
    (+ x y))
;=> 40

My thought was that the biggest challenge devs face is the fact Lisp code is normally written inside out, which tends to create a kind of rightward drift in the code, and lots of parenthesis. So I thought why is that? And it is mostly because of style, in other languages, people tend to flatten their code a lot more using intermediate variables, but in Clojure we tend to just nest arguments in arguments. Even if you use let, that also forces a rightward nesting.

Now I starred at this for a while and I actually started to like it myself. But I realized initially I was going for familiarity, so I even chose a familiar syntax using var = to define the inline variables. But when looking at it for me, someone familiar with Clojure, I didn’t really care for that, but I thought the flattening, or reduction in rightwards drift was even making things easier for me to read. And I realized that’s what most threading macros do already, and those are really idiomatic…

So on my second pass I ended up with this:

(-<>
   <data> [{:id 1 :name "John"} {:id 1 :name "joHn"}]
   <group-by-fn> :id
   (group-by group-by-fn data))
;=> {1 [{:id 1, :name "John"} {:id 1, :name "joHn"}]}

(-<>
 <x> 10
 <y> 30
 (+ x y))
;=> 40

Which is actually starting to feel way more idiomatic to me, and no longer so drastic as with my initial fam macro.

Then I realized that sometimes names are redundant and when you just need to thread the result of the previous thing, they just get in the way, so I extended it to also support:

(-<> (range)
     (map #(* % %) <>)
     (filter even? <>)
     (take 10 <>)
     (reduce + <>))
;=> 1140

This works like an implicit as-> (range) <>, where <> always contains the value of the previous expressions. And now you can mix and match as well:

(-<> (range)
     (map #(* % %) <>)
     <evens> (filter even? <>)
     (take 10 evens)
     (reduce + <>))
;=> 1140

And if you’re annoyed about a thread whose result should always be last or always first, you can nest your choice of threading macros inside:

(-<>
 (range)
 (map #(* % %) <>)
 <evens> (filter even? <>)
 (->> evens
      (take 10)
      (reduce +)))
;=> 1140

(-<>
 <my-string> "a b c d"
 (-> my-string
     .toUpperCase 
     (.replace "A" "X") 
     (.split " ") 
     first))
;=> "X"

And finally, I thought this can all come together as just being the default Clojure syntax, so I extended defn and fn with implicit -<>:

(<defn> lexo [x]
  <rng> (range 1 (inc x))
  <cmp> #(compare (str %1) (str %2))
  (sort cmp rng))

(lexo 13)
;=> (1 10 11 12 13 2 3 4 5 6 7 8 9)

((<fn> [x]
    <rng> (range 1 (inc x))
    <cmp> #(compare (str %1) (str %2))
    (sort cmp rng))
 13)
;=> (1 10 11 12 13 2 3 4 5 6 7 8 9)

And if you want to use #(), it’s easy:

(#(-<>
   <rng> (range 1 (inc %))
   <cmp> (fn[a b] (compare (str a) (str b)))
   (sort cmp rng))
 13)
;=> (1 10 11 12 13 2 3 4 5 6 7 8 9)

I plan to open source this, but I’m looking for feedback beforehand. What do you all think? Is this a good addition to Clojure’s syntax? Does it actually make things easier to read and parse even for experience Lisp coders? And what about non-lispers? Do you think that is still too cryptic for them? Any ideas to improve on the syntax even more? Ideas of other things it should support?

Thank All!

1 Like

Hi, thanks for sharing your thoughts and interesting macros.

Here is my thought about making syntax via macro.

Basically in Clojure, the lexical scope is very clear even working with macros. If you see def you know some variable is created, If you see (xxx [bindings...] ...), you know these symbols will be available for following expressions but limited in this pair of parentheses. AFAIK there’s no the 3rd case and I like how consistent it is.

I heard that Racket have an idea to create a new syntax, there could be a wonderful syntax without S-expression, but I don’t think mixing S-expression with Line-By-Line(don’t know the name of this kind of syntax) syntax is a good idea.

2 Likes

Right, I do think there’s a trade-off here, in that the inline definition do break away from the AST representation, so scope is more implicit. That said, there are precedent for this in the Clojure threading macros I feel, which similarly flatten scope. Maybe the <defn> and <fn> make it more confusing, though it is just that the bindings are just scoped from when they appear to the end of the function, I find that’s still pretty simple. And with -<> it works like anything else, that is, the bindings are restricted to the s-expr as well.

It helps to see what it does under the hood:

(-<> <x> 10
     (println x)
     <y> 20
     (println (+ x y))

Becomes:

(let [x 10]
  (let [<> (println x)]
    (let [y 20]
      (let [<> (println (+ x y))]
        <>))))

Interesting to see your thought process here, and I hope to encourage you to keep tinkering. That said, as just one grey-beard lisper, it’s unclear to me how:

(<defn> lexo [x]
  <rng> (range 1 (inc x))
  <cmp> #(compare (str %1) (str %2))
  (sort cmp rng))

… is more transparent than (or very different from):

(defn lexo [x]
  (let [rng (range 1 (inc x))
        cmp #(compare (str %1) (str %2))]
    (sort cmp rng)))

… which is itself strictly worse practice than:

(defn lexo [x]
  (sort #(compare (str %1) (str %2))
        (range 1 (inc x))))

Naming is famously hard. I don’t think we’d be doing anyone any favors by deploying macrology that encourages everyone to name more things.

On the other hand, this sort of thing:

(-<> (range)
     (map #(* % %) <>)
     (filter even? <>)
     (take 10 <>)
     (reduce + <>))

… has a long history in various Lisps, including Clojure.

4 Likes

You can have a look at https://github.com/Engelberg/better-cond that seems somewhat similar.

I like the flatter syntax, except that is doesn’t work well with static code liners like clj-kondo or joker, which is mostly why I don’t use it.

2 Likes

didibus,

I think that’s amazing for certain situations where nesting becomes very awkward, and efficiency concerns require arrays rather than idioms. Are you planning to open-source it?

1 Like

OK, I’ll be that person…

I think this is absolutely horrible. If you want a different syntax, use a different language.

None of your examples read better than plain Clojure code as far as I’m concerned. You’re just creating a “weird” syntax on top of Clojure – no one is going to be able to read your code except you.

Your example (with -<> and (range)) is just obfuscated Clojure, IMO, and the rest of them are even worse… :frowning:

5 Likes

There is an interesting rationale for flatter syntax at

As a teacher of Scheme and Clojure for many years, I’ve noticed that when newcomers balk at “all the parens”, many times what they are really balking at is the increased level of nesting/indenting in the language. This is especially an issue for people coming from mainstream languages where names are introduced by assignments, which do not increase the indenting level

2 Likes

Ya, actually better-cond was one of my inspirations, but now that you mention it, and I’m looking through it some more, it seems I’m slowly ending in a place much more similar to it then where I started. It actually made me realize I should probably use:

(-<>
  :let a 10
  :let b 20
  (+ a b))

Instead of my weird <a> notation. And if I do, maybe I should call the macro fam again.

I’ve been using a swiss-arrow like -<> in my code as a replacement for as-> which doesn’t force me to pick a name for a while. This one is different though, I’m not sure threading is the right way to put it, and I’m thinking maybe I need to change the macro to be called <-> or something not threading related instead to make that more clear. Because I don’t really thread anything, I just create an implicit binding that always has the result of the last form, and a way to create bindings at the same level as the expression you are in, unlike let.

That makes it a lot more like non Lisp syntaxes, say Elixir.

I think it is interesting you bring this up. Part of my thoughts were trying to rethink idioms and challenge them. What are they really? They’re really conventions that a community creates so you can build familiarity and intuition as to what to expect, so you can more quickly understand code. Sometimes they involve best practices, but I’ll put these aside for now.

In that sense, Lisp idioms haven’t really evolved much I feel since the beginning. Look at Rich Hickey’s’ Java style, it’s a monster. Very unidiomatic, but it used to be idiomatic. But Lisp seems to have just stayed as is since the beginning. In fact, Clojure was the first real challenge to Lisp idioms, making [], {}, #{}, ->, ->> much more idiomatic. So I wonder:

(defn add
  [a b]
  (+ a b))

Why not:

(defn add
  [a b]
  (+ a b)
)

I don’t know, maybe that’s worse. But let’s say:

{:a 10
 :b 20
 :c {:x 100
     :y 200}}

Why not:

{
  :a 10
  :b 20
  :c {
    :x 100
    :y 200
  }
}

Which is how JSON does it.

I’m brainstorming here mostly, but these are idoms, and they got me thinking. In the same vein that I feel Racket2 was trying to question s-expressions. Now, I love my s-expressions. But code like this I have to admit, visually is not very pleasing:

(defn memoize
  [f]
  (let [mem (atom {})]
    (fn [& args]
      (if-let [e (find @mem args)]
        (val e)
        (let [ret (apply f args)]
          (swap! mem assoc args ret)
          ret)))))

Why not:

(deffn memoize
  [f]
  :let mem (atom {})
  (ffn [& args]
    :let e (find @mem args)
    (if e
      (val e)
      (:let ret (apply f args)
       (swap! mem assoc args ret)
       ret))))

I’m no fully satisfied yet though, there’s still too much nesting I think to consider this a worthwhile improvement, not sure.

I wanted a person like that! I actually agree with you to some extent, which is why I was looking for feedback. This macro, its definitely a case of the Lisp curse. But if I ignore the extensions to defn and fn, it can be a pretty restricted use, which could become quite natural, similar to the threading macros I think. Otherwise, its an experiment in familiarity. I wanted to know if you could make Clojure more familiar to non Clojurist, without requiring a new language like Racket2. The question is, if someone starts with this, would they just get confused, or it can be training wheels they can take off later.

Thanks all for the feedback. Keep it coming if you have other thoughts. I’m currently trying to see if I can’t bring some of Elixirs syntax over, specifically, inline do/end. Because now I have inlined let, which reduces nesting, and I’ve inlined the threading macros, because you can just use <> implicitly anywhere, but control structures still create a lot of nested rightwards drift. I’m exploring to see what I can do about that.

2 Likes

This is the amount of rightward drift you get with normal Clojure:

And with my macro, as of now, you get:

1 Like

To my eyes and mind your example is easier to read and follow with the greater indentation changes of regular Clojure. Before I started this reply, I tried to figure out why that is. It could be familiarity, since I have been writing Clojure for a long time now. One thing I see now though, is that the increase in nesting with let can make the code easier to scan. In this particular example, the definition of coll becomes harder to identify with the decreased nesting of your macro.

1 Like

When I say “don’t name things”, that isn’t a Lisp idiom. I would give the same advice in any programming language that made it possible (though it’s much easier to do in expression oriented languages, and is easiest in stack languages).

Lisp idioms haven’t really evolved much I feel since the beginning

WAT?

;; LISP 1.5 code example, LENGTH function. Note that DEFINE is at the
;; top-level *outside* the expression.
DEFINE ((
 (LAMBDA (L)
        (PROG (U V)
              ;; place-oriented assignment
              (SETQ V 0)
              ;; U=L where L is a pointer to the head of the list
              (SETQ U L)
              ;; the "A" here is a label(!), notice explicit
              ;; RETURN and advancing through the list manually by
              ;; incrementing the pointer value in U
              A   (COND ((NULL U) (RETURN V))) (SETQ U (CDR U))
              ;; place-oriented value increment
              (SETQ V (ADD1 V))
              ;; "GO A" is a GOTO to the A label above,
              ;; labels and GO were borrowed from FORTRAN,
              ;; which was the only PL McCarthy had used at
              ;; this stage
              (GO A) 1)) ))

Around 15 years later, the same program in scheme compiling to a program of equal efficiency :

(define (len l)
  (if (empty? l)
    0
    (+ 1 (len (rest l)))))

It looks like the idioms advanced during that interval, no?

There’s an odd ahistorical belief among younger programmers that Lisp was handed down from the mountain in a single burst of divine insight, but that is absolutely not how it happened. Hell, even in my own relatively brief 34 years of Lisp programming there have been many change in what’s considered idiomatic among various dialects.

This is the amount of rightward drift you get with normal Clojure

Yes, and the x position on the screen is part of what communicates what’s happening in the code. For more on harnessing the visual cortex to improve code readability, see:

5 Likes

@didibus,

I’ve got a few comments I’d like to share.

  1. I do think there’s value in preventing the “rightward drift” you describe. Thanks for the screenshots, I didn’t see what you meant until you posted those.
  2. I subjectically strongly prefer the :let syntax to the <a> = 123.
  3. By preventing rightward drift, you pull an expression-based syntax towards a statement-based syntax. I could no longer select by matching parentheses and have something that made sense.
1 Like

One of the hidden powers of lisp (for me at least) is how easy it is to manipulate code at an expression level. Most other languages force you to think of codes in terms of lines, but lisp is a tree. This makes it really easy to build text manipulation tools on it like paredit etc. Since moving to Clojure I can’t remember the last time I had to select/highlight something to manipulate it. Just sp-copy-sexp. If you move away from the expressions it pushes this sort of tooling further out of reach. Not that you can’t have expression based code manipulations in other languages, it just becomes much more complicated to implement for very little gain.

You can do really powerful things. The other day I was writing some Elisp and it dawned on my I could get emacs to manipulate the source code of my Elisp functions. The code could move itself.

((bar)
 (transpose-sexps -1))

If you evaluate (transpose-sexp -1) in this context the expression moves itself, in your source code.

((transpose-sexps -1)
 (bar))

Sure this is a silly example, but that sort of power is something I wouldn’t want to lose to avoid rightward drift. Still it’s always interesting to explore these things.

1 Like

I’ll second Jack on the “encourage you to keep tinkering” sentiment, but I specifically and explicitly want my let bindings to be s-expressions. I want the scope to be visible. I very distinctly don’t want implicit scope.

I also don’t find rightward drift that annoying on modern screens, nor do I find it overly difficult to counteract when it does get excessive.

I’ll also voice the least-meaningful objection, which is “please don’t add syntax”. I appreciate that Clojure has so few special syntactical operators (well…outside of destructuring), and introducing a new one like a top-level :let or <ad-hoc introduction of names> needs to clear a very, very, very high bar of utility and elegance for me to even consider it as a possibility. Yes, this objection is extremely close to “I like things how they are and don’t like change”, but I think it’s distinct enough to add to the conversation.

I think I would be sad and frustrated if I had to read Clojure code that used this kind of macro.

I wanted to know if you could make Clojure more familiar to non Clojurist, without requiring a new language like Racket2.

I’d prefer that this non-Clojurist learn lisp syntax. I think we overestimate how much the parentheses are the hard part of Clojure, and underestimate how difficult it is to switch to a functional approach using a composition of small-to-medium libraries (not frameworks, not microlibs like in modern JS-land) and immutable data structures in a hosted language over a complex ecosystem like the JVM or Closure/JS. I think Racket, as a teaching language, has a clear reason to propose paren-less Racket2. I don’t see the appeal for Clojure, but if we did go that route I think I would prefer a separate language in the style of Racket2.

4 Likes

I think this articulates why my reaction was so negative. Clojure is expression-based and that’s a simple “rule” to learn – everything is an expression – but in trying to make Clojure look more familiar to non-Clojurists (i.e., less Clojure-y) you’ve undermined one of the fundamental simplicities of the language.

I also agree with those folks who pointed out that this new syntax is no longer amenable to the structural editing tools that we get used to for everything else in the language.

I’m glad to see I’m not alone in finding this additional syntax to be less readable.

Edit/p.s.: I should probably also confess that I do not find better-cond to be “better” than cond: I find most of the examples used to be less readable than the alternatives. For me, the nested structure aids readability by delineating clear “scopes” in the code – and if you find your code is drifting too far to the right, then you should probably refactor into smaller functions to make things more readable/more self-descriptive.

4 Likes

Rightward drift is perhaps worth minimizing but at what cost? Even if the code in the second image is a clear cut aesthetic winner (I personally don’t think so), you’re using an external dependency and you’re no longer using standard clojure code that can be understood by any clojure programmer.

(I’m going to focus on better-cond, mentioned above, because this is one I’ve actually considered using a lot, but all of this applies to your threading macros as well)

I think the better-cond library makes some good decisions towards this end in that it still ends up looking somewhat like core clojure code. To me it feels consistent with the the clojure.core/for options. But does it justify bringing in a dependency that almost definitely doesn’t help solve your problem, but merely in the arrangement of your solution?

I’ve been aware of better-cond for some time but haven’t actually used it yet. Maybe once per month I find myself in a situation that reminds me of it. Usually an elaborate cond statement with various let bindings and threading based on the conditionals. This usually occurs in the “first draft” of coding phase, the initial expressing of the thing as code. Sometimes these hairy situations are unavoidable but more often than not after some thought and time, they naturally disappear; the code was broken up into separate functions, or the whole concept ended up being tackled with a different approach.

Even when the hairy situations stay hairy, I think that’s a good thing to just leave them that way. I think it’s good to have some ugly code safely quarantined inside some function somewhere that you know about. Maybe one day it can be looked at again and you’ll think of a way to make it good. Or maybe it will just stay there forever. I think it’s better to have one or two of these per thousand lines of code than the alternative.

The alternative is to depend on better-cond at the first sign of hairiness. Now your first draft mediocre code looks and feels good! And you move on to the next function you need, and then hey why not, we can save a few keystrokes with better-cond so let’s go for it. And you continue in this manner. But you never catch the fact that your code would have looked hairy because it WAS hairy! Something was wrong with it, it wasn’t properly simple. You start to accumulate things until everything is a giant hairball.

I don’t want to be disparaging. The better-cond library author mentions this in their readme that it is ingrained in their style of code and they always use it. I think one of the best parts of lisp is that this type of thing is possible. You can bend clojure to your will, you can morph the language to something you like even better.

But personally I prefer to defer to Rich’s language design sensibilities over my own.

2 Likes

Being forced to get use to Clojure syntax is a great exercise to understand the LISP power. It’s also a great filter for whom would like to stay in their comfort zone of common procedural languages syntax.

I’ve also tried to investigate how to transform python like syntax to avoid parenthesise when I begin Clojure. Soon realised it’s really hard to make it better than the original syntax, if not impossible.

If you want to sprint, train like sprinters instead of jogging training.

2 Likes

Edit: from the same author

v23 of FARGish: Restarting from scratch, this time in Python
v22/ contains the last attempt in Racket, abandoned. Also includes old
Clojure code from still earlier versions.

Source: GitHub - bkovitz/FARGish: Generic tools for making FARG models

:face_with_head_bandage:

Hum, that’s true, representing things as AST will make the scopes very clear in all cases, but sometimes also difficult to visualize a lot of it. I could have done the same in the macro as well, and then the two would have been much similar, in that the thread would not be inlined, but only the lets would.

:bowing_man: :bowing_man: :bowing_man:

That’s awesome. Glad to have someone with so much history and experience of Lisps around.

Gotta admit, I can only consider myself a Elisp and Clojure programmer, and superficially have some knowledge of Common Lisp and Scheme, mostly from reading, not a whole hands on. Did not think to go further back to 1967’s Lisp 1.5 :wink:

That’s a great point. I hadn’t quite considered the age of Lisp to be honest. It’s a very old language, and the idioms already iterated quite a lot from even before I was born :stuck_out_tongue: So not seeing them change in what can be considered recent Lisps, like CL, Elisp, Scheme, etc. could just be because they’re already pretty hardened and battle tested.

That paper looks great, will need to read it.

Ya, I’m pretty settled on the :let instead of the <a>. The :let also allows me to support de-structuring like :let [a b] [10 20]

Yes that’s a good point. The AST expression syntax lends itself perfectly to structural editing. The more flat syntax is better for Vim like line based editing.

It seems after a lot of experimentation, I’m almost more confused why people find the Lisp syntax so strange and hard to read. I’m trying to see if it’s that I need to flatten things even more. My guess is, it’s only familiarity, and that’s why I had called the macro fam at first :stuck_out_tongue: To bring something more familiar to non Lispers.

I tried some more things like:

Current macro:

:let x 1
(cond
  (= 1 x)
    (:let a 100
     :let b 30
     (+ a b))
  (= 2 x)
    (:do (println "= 2")
         (* 100 2))
  :else
    (throw (ex-info "Failed" {})))

Normal Clojure:

(let [x 1]
  (cond
    (= 1 x)
      (let [a 100
            b 30]
        (+ a b))
    (= 2 x)
      (do (println "= 2")
          (* 100 2))
    :else
      (throw (ex-info "Failed" {}))))

Proposed macro:

:let [x 1]
:if (= 1 x)
  :let a 100
  :let b 30
  (+ a b)
:elseif (= 2 x)
  (println "= 2")
  (* 100 2)
:else
  (throw (ex-info "Failed" {}))

And now it’s just starting to look more and more like an imperative statement based language. And what’s happening is that as I try to make things more familiar, I’m even having to recreate constructs like if/else. Because without the nesting, you need keywords to indicate the distinct parts of the code. One could call these keywords statements. They state the semantics of what follows, they state where things start and end, etc.