What's the value proposition of macros?

Community Center Watercooler
1

I realize that macros are one of the definitive "secret weapon"of Lisp, and allow compile-time rather than run-time effects. However, lately as I’ve worked with my students I’ve been painfully aware of two major costs to them:

  1. Macros introduce new mental models of the project space, requiring cost comprehension efforts

  2. They play poorly with the REPL, breaking the REPL-driven experience that is truly one of the core features of Clojure.

Now, perhaps I just have a lack of understanding on 2. But in my experience, you can’t just reevaluate the macros to effect a change, but have to reevaluate everywhere the macro was called. This breaks my efforts to emphasize simplicity to my students, and is one of the reasons I have found it so pleasant to move from macro-heavy Compojure (as well written as it is!) to Reitit, as just one example.

So, what can macros give that is worth these two (major!) costs?

2

As far as I understand the idea, they give you (runtime?) code generation basically without cost, when you compare them to the mental and technical overhead of some other forms of this concept (looking at you, Java annotations).

Isolated from the rest of the language, their value proposition might not be as apparent. They’re defined using regular Clojure syntax, in regular Clojure files, receiving the ‘code’ they work on in plain Clojure data structures. Of course I’ve left out the (little) extra syntax and the requirement for a basic understanding of LISP macros, but those are small and even smaller compared to other macro-like systems, given that you can basically extend the language (and syntax, to an extent) however you like without waiting for vendor/upstream changes. This is what I consider their value proposition to be: allow me to write a procedure that takes code, and generates code for me, and make that as easy to work with as possible, while being integrated into the language as well as possible.

For example, as I understand it, the or macro and similar “code path” affecting macros (cond, case, …) are/can be easily optimized to generate code that doesn’t call into forbidden or unnecessary paths, and have that code ready to be optimized even further by the JVM.

In response to your pain points, I agree with your sentiment. I’ll not act like I know anything remotely in-depth about all this, but I’ve experienced your problems as well and I think while frustrating, they’re nothing compared to the frustration I’ve had with Java annotations or C-style macros. My issue with Java annotations is not even consuming them, like writing an @() somewhere, but writing new annotations is … well. Even just using them has problems, much deeper and more final, in the way that it’s basically just string magic.

So, yes of course there’s more mental overhead, but little compared to similar tools for what macros do, or rather, what they enable a core, library and commercial developer respectively to do as code consumers/producers. Yes, there’s can be some hassle when working in the REPL. But macros - in my personal opinion - shouldn’t be used liberally for exactly these reasons. They’re an immensely powerful way to hook into the language itself, but that comes with a bit of cost.

Having a few, well-placed macros to “neaten up” boilerplate-heavy code sites, or consuming them from libraries, seems to be a strategy here. I don’t think macros are a good go-to when implementing solutions in Clojure, and I think they’re an unnecessarily complex thing for newcomers to spend time on (especially in Clojure, where the “simple stuff” really matters, and there’s so much in clojure.core to explore).

Like everything in this language, it fights you while you’re “doing it wrong” :wink:

1 Like
3

Thanks for your thoughts. I admit that I’ve not attempted anything macro-like in other languages (macros in emacs/word processors being a different type of thing), and a part of my bias is surely based on the fact that I am continually jumping between code-bases (sometimes 3-5 different programs in a day) and that cognitive cost, combined with a lack of dedication/critical mass to make macros worthwhile for those code-bases, colors my bias.

That whole “don’t have to wait for language designers to create XYZ” line is at the head of every write-up on macros; nonetheless, I’ve never actually been in the situation of wanting something like that, despite working in a half-dozen languages on a daily basis (which, as an aside, is one of the things I’m trying to use full-stack Clojure to curb). Maybe this points me to some directions in which I can improve myself as a programmer.

4

Regarding the “don’t have to wait for language designers to create XYZ” thing, I think we’re in this situation all the time! Maybe not so much in the role of a macro code producer if you’re working in the industry, but as a consumer most likely (like compojure's defroute or similar, syntax that wouldn’t exist in core). If you’re a library author, it’s a different story, and for a Clojure core developer, another.

Personally, I’ve been in a situation where an architecture problem somewhere deep (entirely unforeseeable, of course) required me to basically rewrite a large portion of the code. Fortunately, a simple macro took care of most of the pain while not touching the architecture beneath it at all (macros can be incredible), and although it’s been a thorn in my eye ever since, it’s become somewhat of a motivating flaw. So I’ll probably never rewrite that architecture under it :smiley:

2 Likes
5

My experience is that macros can be very useful for improving syntax/boilerplate and there are a few cases where you need a macro – a function just won’t do. Overall, however, macros do not compose well (like functions do) so they should be used sparingly.

For example, at work, we have 61,000 lines of source code containing just over 3,200 functions, and we only have 42 macros in all of that. We also have 20,000 lines of test code – and we use macros more heavily there to reduce boilerplate in tests (lots of with-xyz macros to help provide context and localized fixture-like behavior for tests).

2 Likes
6

Well, lets start with a clear statement:

  • Don’t be writing a macro if you don’t understand its value proposition.

Think of it like reflection in Java, its a tool of last resort, because of what you mentioned, macros don’t compose well with each other and with functions. And yea, you have to recompile code if you’ve changed the macro, a reloaded workflow would help in that regard.

Now, where it comes to the value proposition is in terms of syntax sugar. You need to think of order of evaluation.

Clojure uses an inside out, left to right, argument first, evaluation order.

(<function> arg1 arg2 arg3 ...)

When the above form is evaluated, arg1 is evaluated first, then arg2, then arg3, etc. Afterwards, the values returned by the evaluation of each arg is passed to the function for evaluation. This should be intuitive to everyone.

This in turn is the basic flow of Clojure, and most modern languages.

Now how do you control that flow? How can you modify this order of evaluation?

Other languages say you can’t. They give you premade alternative order controls, like if, case, for, while, ||, &&, etc.

Lets look at if.

(if condition then else)

This is not the nornal evaluation order/flow we saw before. Condition is evaluated first, and only one of then or else will be evaluated, and we’re done. If it was evaluated normally it would evaluate condition followed by then followed by else and finally evaluate if giving it the results of condition, then and else.

Now, you need some primitives to use in the beginning to start altering the flow, and in Clojure they’re given to you as special forms. But this is nornally where other languages stop. In Clojure, you can make new evaluation order and flow altering macros.

This means you can build cond, case, when, if-let, condp, if-not, for, while, doseq, dotimes, etc.

You might think you’re not getting a lot of mileage out of macros, but that’s because clojure.core already wrote most of them for you :stuck_out_tongue:

In my opinion, customizing the order of evaluation, or the flow of evaluation, is one of the best value proposition of macros.

Other notable uses:

  • Rearranging code, such as ->, ->>, doto, etc.
  • Adding dynamic state, such as with-open, time, with-in-str, etc.
  • Wrapping things for you, such as delay, future, lazy-cat, etc.
  • Building DSLs, such as ns
  • And adding or removing code, such as comment, doc, assert, defn-, etc.
9 Likes
7

This is a great answer! Thank you. It helps to have some concrete examples of adjusting flow-of-execution.

8

Don’t forget that the core.async Clojure library relies heavily on macros. It rewrites code that is written as if it were synchronous into asynchronous code and uses macros to accomplish that, I believe. Doing that would probably be impossible in most other languages.

2 Likes
9

To me macros are part of a larger story related to syntax and programming paradigms. I wrote about this here: So yeah, about Clojure syntax, pasting the outline here:

  1. Does syntax matter?
  2. The ingredients of Clojure’s syntax
    1. Data literals
    2. Macros
  3. Consequences
    1. Verbosity is a solved problem
    2. Separation of concerns: code layout ⊥ program structure
    3. Code = Data = Data Viz
    4. Tooling as libraries
    5. An ‘all-tracks’ language: embedding paradigms
    6. Saner language stewardship
  4. Summary

IMO, if programming was building a road network, then writing a macro would be like building a bridge or digging a tunnel: not something you want to do most of the time, but which can sometimes give you access to otherwise hard to reach places.

1 Like
10

I wrote about the topic in depth here: https://yogthos.net/posts/2017-10-03-MovingForwardByLettingGo.html

To sum up, macros allow extending the language in users space. This means that you don’t have to bake new features into the core of the language allowing it to stay small and focused. When features are added via libraries they can be easily sunset as usage patterns change while projects using these libraries will continue to work fine. However if features are added to the language itself they end up introducing bloat over time. Many popular languages today have a lot of legacy features that aren’t considered to be best practices, but are now part of the language because that’s how it was used at the time.

1 Like
11

Thanks; this “new features to the language” case is the most
frequently cited benefit of macros, but I have never, in my years
of programming, actually said “I wish this language had
XYZ…”. However, considering the examples I know in your blog
post (Compojure, Specter), I think I see where my experience comes
up short: I have yet to tailor my products to a domain. In
essence, I don’t do much writing for developers, which seems to be
where macros shine – instead, I write applications for end-users
who know nothing about code, and less than three people will ever
be developing/maintaining the codebase at any given time. So this
is my take-away: macros allow you to better fit your program to
your domain, and where scale of the project doesn’t reach
domain-specific levels of consideration, macros may be too
expensive in terms of complexity-cost.

It is also interesting to me to consider the difference between
Compojure’s use and Specter’s use of macros. Because Compojure
uses macros to define program constants which need to be in
frequent use yet also iteratively and dynamically developed, I
find the macros to be slighlty obstructive to the development
process. However, because every macro in Specter is most likely
used more like a lambda (every one is somewhat unique), this
limitation to the REPL-based workflow isn’t encountered as
much. I’m on the verge of saying something profound about what
that means for the pragmatics of macros. Something about macros
that define things vs. macros that perform functions. Give me a
minute…

12

Yeah, I think that as a rule macros are best reserved for library APIs where a library expresses a particular domain that can’t be easily expressed given standard Clojure semantics. Such cases tend to be fairly uncommon in my experience.

I tend to have very few ad hoc macros in my code, and the times that I do I keep their usage confined within the namespace. That way you’re not leaking the semantics into the rest of the project.

Another use case for macros that’s worth noting is performance optimization. For example, a friend of mine was working with Google protocol buffers, and he needed to generate the classes that represent the schema. Doing it at runtime was too expensive in that case so he wrote a macro that would generate these classes at compile time. It’s not a use case that comes up often, but it’s nice to have a tool for optimizing performance when you need to.

So, my general advice would be to avoid macros unless there’s a really good reason to use them over functions + data, but it’s good to be aware of them as they are a very powerful tool.

1 Like
13

Macros let you extend language syntax in a library (don’t need to add special forms to core, like you would to add this to javascript)

async/await: http://funcool.github.io/promesa/latest/#async-await-syntax
core.match: https://github.com/clojure/core.match/wiki/Basic-usage
go blocks: https://github.com/clojure/core.async/blob/master/examples/ex-go.clj

14

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.