Why is the macro systems in lisps considered so valuable

Community Center Watercooler
1

In http://www.paulgraham.com/avg.html, Paul Graham explains why Lisp is a better language.
I interpret him as saying that the macro-system was very important for them, and that 20-25%
of their code was macros.

Having worked with Clojure for a while now, I think I’ve yet to write a macro. What I have done though
is to rewrite macros into functions.

So my thing is that while I can see that macros have their place in extending the language without extending
the core of the language, eg core.async, and I could easily write an unless if I felt I needed that, I could never
justify doing either of them, the former because it would take too much time if I ever succeeded, and the latter because it wouldn’t provide enough value.

My questions are then:

  1. For a normal application developer, what is the power that macros bring to the table?
  2. Have you written macros that without, your app would have been impossible, taken significantly longer to develop or otherwise brought value that you couldn’t have gotten in other languages/without macros?
6 Likes
2

The power of macros is the ability to define new syntax, and to automate the generation of code. As such I don’t think I’d say any uses of macros are required or enable something that is otherwise impossible. But this distinction is true for Clojure and any high level language.

Is it impossible to write immutable data structures without a GC? No, but a GC makes it a ton easier. Is it possible to write functionally pure code in a OOP language? Sure Clojure is written in Java, but go look at the source and you’ll see how concise Clojure is compared to Java.

So I’d say macros are about doing more with less code, and in reality they’re just a natural extension of Lisp being so close to an AST to begin with.

6 Likes
3

I wrote on the topic here in detail. To sum it up, I think the main value having powerful userspace language extension is that it allows the core language to stay lean. Usage patterns change over time, and what’s considered best practice changes over time.

When a language can’t be extended in user space, then it becomes necessary to add features to the core language itself. You can see the effects of this with languages like C++, Java, and JavaScript where usage patterns changed significantly, and the languages ballooned in size and complexity.

Clojure usage patterns have also changed over the decade of use, however the core language has stayed remarkably small. I think this is largely due to the fact that most things can be expressed as libraries, and when a better idea comes along people can just stop using the library.

Meanwhile, projects that depend on this library will keep working just fine, so you don’t have the problem of breaking backwards compatibility by cutting language features.

  1. For a normal application developer the primary benefit comes from having a smaller and cleaner language.

  2. There’s nothing you couldn’t do in principle without macros, but there are many situations where macros have made code in my applications cleaner and helped abstract domain specific logic.

10 Likes
4

It lets you create new abstractions if you need them. Which can sometimes make the code cleaner, easier to read, more concise, easier to reason about and with less bugs.

It’s not the only way to create abstractions though, you can use functions, data, records, multi-methods, protocols, etc. That is, it is a very powerful way, and can fill the gap left by the other mechanisms when nothing else can do.

Also, keep in mind Clojure is batteries included. Macros don’t just mean its easier for you to create powerful abstractions. It’s true of libraries as well. That’s why the core team itself can provide us so many of them, as well as the community. Otherwise, the core team and community would take way longer building them, and Clojure would have half the features it has by now.

Remember how small Clojure’s core team and community is? Yet it equals and often surpasses languages with way more man hours behind them in feature set. That’s in part thanks to Lisp simple and powerful macros.

Yup, I have. It doesn’t happen often, but when it does, its fantastic.

I wrote a macro that maps clojure data to java pojos and back. It helps us interop our systems together, since we have mixed java/clojure code bases.

I wrote a few macros to extend spec. One that validates spec existence, and another that adds support for speccing closed maps.

Those are three that come to my mind. But I think there’s a few more. I’d say we come up with 2 to 3 useful macros per year.

4 Likes
5

I can vouch that interop is a phenomenal use case for macros - I once built a small clojurescript macro that allowed the mounting of clojurescript reagent components into plain javascript sections. Using a macro allowed me to abstract the hacky mount interface so that defined components were defined exactly like reagent ones.

The plan here was that as the code moved between javascript to clojurescript, components can be rewritten independently of the reagent mounts apps, and as reagent took control over those parts, the components that were already rewritten with the macro could easily transfer over. A code base rewrite could then safely be staged in small independent parts.

2 Likes
6

I tend to concatenate a data conversion pure function (pipeline).

Macros can be a simple, violent, buggy, hard to read, difficult to debug and observe. it’s temporary solution.

Macros are suitable for language extensions, and application-level development is best not to use them.

DSL usage is code conversion, Using data style representation is better than using function style representation. A series of pipeline functions are concatenated to form a compiler for converting DSL data into target code and then evaluating it.

The best abstraction is: data and logic are strictly separated, data-flow is current-flow, function is chip, thread macro (->>, → etc.) is a wire, and the entire system is an integrated circuit that is energized.

2 Likes
7

Another macro power not mentioned here is that they are executed at compile-time. Functions are executed at run-time.

We use this distinction in our multilingual web application to extract the text that needs to be translated by simply compile-ing it: the macro code fills an atom at compile time, then we have a boot task that compiles our code (doesn’t run it) then dumps the atom content to a file. The value of this? We don’t need to maintain a dictionary of translation in our app because it’s just too cumbersome, error-prone and the app and dictionary tends get out of sync quickly. We send the file to an external translation service and we know for sure that we send only what’s needed, nothing more, nothing less.

That’s the only macro I’ve ever written but it’s pretty useful and is a good use case for them. If you want to learn more about macros I recommend reading the book Mastering Clojure macros that details how they work and showcase all their use-cases (it’s more than for DSL and language extension).

4 Likes
8

My project also supports multi-languages (Chinese, English, Chinese+English), some of the multi-language strings are placed in the code, some are placed in the DB, and the “multi-language processing pipeline functions” can be concatenated where needed, It handles these situations simply and perfectly.

According to the current popular middleware terminology, this is the middleware of data flow style. Flowing is data, not function. It’s simpler, readable, easy to debug, test and observe

2 Likes
9

I agree with you. Data abstractions are better when they can do, and you should favor them always.

My use cases for macros are in fact language extension use cases. And our macros are never application specific. They live in seperate packages, which are used by different applications.

Like the clj->java and java->clj macros to go from object to clj and back. Without extending the Clojure semantics, such a conversion is many more lines of code, and much harder to read, and easier to accidentally make a mistake.

Or I have a try/catch extension macro that allows multiple exceptions to be caught. Again, when dealing with Clojure only, you can branch your catch code on ex-data, but when dealing with a lot of Java exceptions, such a macro does improve code quality and readability. Otherwise you’d need to have a cond clause on exception type within your catch block.

In most cases, you can think of macros as code generation functions. Every piece of code that is repetitive and verbose adds extra potential for typos, or mistakes when writing it, and increases the burden to read it, by adding a lot of white noise. A macro can be useful in such case, at least in my humble opinion. That’s the case for my clj->java macro

Other times, you want to extend an existing macro, like my extended multi-clause try/catch. Or my extension to spec, or multi-clause when-let (https://dev.clojure.org/jira/browse/CLJ-2213).

So, I think you point out a great point, you really shouldn’t have application specific macros, I’m not sure I can think of a good reason too, in such case, it feels data or functions or something else can probably be used more effectively. But, it does come in handy to be able to extend Clojure sometimes, and the extensions can help you more effectively build applications.

I think there are language extensions that, if standardized into Clojure core, requires a much more careful process, because you commit to new patterns that can never be changed ever. And so, until you know there are no edge cases, or better patterns, or inherent flaws with the pattern, it probably shouldn’t be made standard. But, on a team, these burden are reduced. A team can more easilly adopt new patterns even if they have some flaws, its easier to be aware of them and work around them, or pivot away from a pattern that didn’t work out as well as it should have. Or, within the context that the team uses the pattern, those flaws might not matter, and thus the pattern is a net positive, but in arbitrary context it wouldn’t. For all these reasons, I believe its a great strength of Lisps to allow teams to develop their own extensions.

2 Likes
10

In general, I don’t recommend assessing programming language features in isolation; how they play together is much more important. In particular, I think the story behind Clojure macros is incomplete until you consider how they relate to syntax. I tried to describe this story here: http://vvvvalvalval.github.io/posts/2018-01-06-so-yeah-about-clojures-syntax.html

It’s totally normal that you would not need to write a macro as an application developer; that’s more for library authors. The benefit is usually not to help you express one particular aspect of your domain logic; rather, it’s to eliminate accidental concerns in your code, and at the ecosystem level to make a lot of powerful tools available to you at a low cost.

2 Likes
11

Hey @slipset,

Great question.

I worked in Common Lisp for a while, and I used macros a lot. Paul Graham’s argument made a lot of sense to me. But in Clojure, I use macros way less. Like almost never. I’ve wondered why ever since.

I think it’s mainly because Clojure is more functional that Common Lisp. Common Lisp had first class functions and recursive data structures, but everything was mutable and a lot of the constructs were procedural. You could say it’s even slightly lower level than Clojure. I think a lot of the macros I wrote were to automate/abstract the procedural parts into something more “declarative”.

However, in Clojure, we’re closer to a lambda calculus (in some ways, but ways that matter). We do much more in data pipelines and map/filter/reduce. That stuff was done in Common Lisp, but less often. People leaned on big macros like LOOP to sum, average, count, and aggregate data. Macros seemed like a great answer. You could do all that with a few symbols, and only run through the list once. We don’t care so much anymore how many times we go through the list. And we prefer to be able to write sum as (reduce + 0 ls).

There’s another reason, and I think it’s more about the design choices of Clojure. Clojure’s syntax is more expressive than CL’s. Take, for example, the literal data structures. You could write a function like Clojure’s hash-map in CL, but most people did it with a macro. The idea was “why do this at runtime when I could do it at compile time?” But it’s more than that. If you look at CL books, they’ll often model stuff with data (lists and cons cells, mostly) and then think “this is syntax, it needs a macro”. I think we’re more content to just use data and parse it at runtime. Is it because we have bigger machines? Maybe. But it’s also a deeper idea of data orientation. If it’s data and it’s parsable at runtime, we can maybe manipulate it at runtime, too.

Clojure also comes with a lot of the syntax/macros people were writing in Common Lisp. Take the #(...) syntax for functions. That didn’t exist in CL, so there was a lot of macros people would write first thing to save keystrokes. People wrote a macro that let them do this: ```
#L(if (oddp _) (* _ _) 0) Those underscores are the argument. Paul Graham talked about aif (“anaphoric if” that would bind the it variable). It’s similar to if-let, which Clojure graciously provides. I think there’s a lot of stuff built in that you would want to use on your own.

Clojure is more data oriented. CL (and Racket) is more syntax oriented. It’s a community orientation, facilitated by the affordances of the language. We tend to think of everything as a data problem. Racketers tend to think of everything as a language problem. Just different perspectives.

Rock on!
Eric

11 Likes
12

Can you elaborate on this one?

1 Like
13

Yes! Common Lisp has a macro called LOOP. It’s actually complicated enough to call it a DSL. For example, you can do this:

(loop for i in *random*
   counting (evenp i) into evens
   counting (oddp i) into odds
   summing i into total
   maximizing i into max
   minimizing i into min
   finally (return (list min max total evens odds)))

This is going to count two things, the evens and the odds. It sums them into total. It finds the max and the min, and returns them all in a list. It does that in one iteration.

We don’t do that in Clojure. We would just iterate through the list multiple times.

(let [evens (count (filter even? numbers))
      odds  (count (filter odd? numbers))
      total (reduce + 0 numbers)
      max (apply max evens)
      min (apply min evens)]
  [evens odds total max min])

We don’t mind iterating five times. And we don’t mind that we can’t just say “summing”.

2 Likes
14

I see, but I’m not sure I agree that we don’t mind. That’s the whole reason behind lazy sequences and transducers.

Both of those are performance optimization strategies, while also being abstractions. The former will only process the data that is actually required. The latter will perform loop fusion, thus looping only once over.

But I understand your main point, we care less potentialy, at least not prematurely and at the cost readability, and composability.

15

@didibus @ericnormand

Alternative:
Element level separation of data and logic, data stream processing.
Don’t use transducer unless absolutely necessary. It’s like middleware, too complex, difficult to debug and observe.

(defn f [[evens odds total amax amin] x]
  (let [[evens odds] (cond 
                       (even? x) [(inc evens ) odds]
                       (odd? x)  [evens (inc odds)]
                       :else     [evens odds])
        total (+ total x)
        amax  (max amax x)
        amin  (min amin x)]   
     [evens odds total amax amin]))

(reduce f [0 0 0 ##-Inf ##Inf] [5 6 8 -3 -9 11 156 6 7])

;;[4 5 187 156 -9]
1 Like
16

Yes, I think for this example, you’re right. A reducing function of that sort is best. If you needed the performance, otherwise juxt or a let like @ericnormand wrote I feel are simpler.

That said, for the curious, Xforms provides additional transducers pre-written for you which can do this kind of thing too. Using transjuxt.

17

I don’t like to memorize a lot of data forms and functions. The functions of clojure.core is just right. I usually use it to manipulate data, organize code in a sequential structure of dataflow, keep every step debuggable & observable. For beginners of niche language, It is the most simple.:smile:

1 Like
18

This is an old thread, but there is one point that I don’t think has been made here yet: Whether Clojure macros are needed often (they’re not, for me), or rarely (but sometimes make a big difference), lisps are not unique in having some sort of macro language.

What’s different is that in lisps you have the full power of the language itself to manipulate language constructs in their original syntax.

Contrast this for example with C, which has a completely separate macro language (which isn’t very powerful), or Javascript, where you can build and evaluate any kind of code in strings (which is error prone).

1 Like
19

Here is a very simple use of macros that I came across recently.

Say you have a function like this:

(defn greet [user]
  [:div "hello " user])

You want to use it from both Clojure (backend with Hiccup) and ClojureScript (frontend with Reagent). The problem is that for Hiccup you have to call it like this:
(greet "some user")

and for Reagent you have to call it like this:
[greet "some user"]

But you’d like to write your markup code just once for both backend and frontend (in CLJC). One way to do it is to wrap this call in a macro, we’ll call it ‘cc’:

(cc [greet "some user"])

And this ‘cc’ macro will rewrite your call at compile-time, depending if you’re using it from Clojure or ClojureScript. This way you can have a library of common UI components that you can use from both frontend and backend.

;; for Clojure
(defmacro cc [args]
  `(~@args))

;; for ClojureScript
(defmacro cc [args]
  `~args)

The fact that this conversion is done at compile-time, means there is no runtime penalty for using it.

1 Like
20

In Clojure, macros are implemented as a special function, unlike Lisp, without this effect.