Create code-like data structures dynamically

Questions & Help General Questions
1

Is there a comfortable way to implement a mechanism that takes a list, searches for the keyword clj and evaluate what is inside the clj?

Here an example:

'(a (clj (+ 1 1)) b) => '(a 2 b)

My first try was a function eval-clj, that goes through the list and calls eval for the clj expressions. But that is not practicable, because the eval would not be in the right context. The example below would not work, because the variable x is not visible. There seems to be no way to achieve this with functions.

(let [x]
  (eval-clj '(a (clj x) b)))

So I try it with a macro.

(defmacro eval-clj [expr]
(list 'list (list 'quote (nth expr 0)) (second (nth expr 1)) (list 'quote (nth expr 2))))

(macroexpand-1 '(eval-clj (a (clj (+ 1 1)) b)))
(eval-clj (a (clj (+ 1 1)) b))

It is just a small example. But it looks to me very complicated. I have to move the quotes from outside to inside. I would have to do it recursively.

And the macro solution does not handle dynamic things. This would not work:

(eval-clj (clj (map identity '(a b c))))

I understand why this does not work and why it is so complicated. But is there any idea to do such things? My use case is a domain specific language for something which is outside of Clojure, where the code is created at run time.

It would be very nice if my DSL could look like code. For example it would be nice to use expressions like '(schedule-task some-task-id) without the need to actually defining these symbols.

I have the impression that what I want is theoretically impossible. Are such things may be possible in other languages like Racket? People say often that Racket is good for language designers.

What I find strange that the following things work

(clojure.edn/read-string "(set-name \"Sarah\")")

(let [name "Sarah"] (str (hiccup.page/html5 [:p "name: " name])))

while for our own data structures this is not possible:

(let [name "Sarah"] '(set-name name))

The workaround below does not look much like a DSL.

(let [name "Sarah"] (list 'set-name name))

I don’t know, if everybody can follow me. If nobody has an answer, that is ok. I just leave here what I found out in case someone finds it interesting.

2

There’s no need to make it a macro.

You can implement it as a plain function that uses eval and clojure.walk/prewalk:

(require '[clojure.walk :as walk])

(defn eval-clj [form]
  (walk/prewalk (fn [item]
                  (if (and (list? item)
                           (= (first item) 'clj))
                    (eval (second item))
                    item))
                form))

It computes the result right away, evaluating the (clj ...) forms right as it walks the whole structure:

=> (eval-clj '(a (clj (+ 1 1)) b))
(a 2 b)
3

@p-himik This does not work unfortunately. See this example.

(let [x 2]
  (eval-clj '(a (clj x) b)))
4

Values from let bindings are not available to forms passed to eval.

But you also can’t use a macro for it since, even though macros have access to &env, they don’t have access to the values themselves.

So, I don’t think you can do exactly what you want in Clojure, at least not without contorting the language into a shape it’s not supposed to fit.

5

This won’t help you with code sent in at runtime, unfortunately. (See my next answer for how to do that.)

You should probably ignore the below and see my next answer

OLD ANSWER:

For code available at compile time you could do something like this:

(require 'clojure.walk)

(defn calling?
  [sym form]
  (and (seq? form) (= sym (first form))))

(defn quote-unless-clj
  [form] 
  (cond
    (calling? 'clj form) (second form)
    (calling? 'quote form) (quote-unless-clj (second form))
    (seq? form) (apply list 'list (clojure.walk/walk quote-unless-clj identity form))
    (coll? form) (clojure.walk/walk quote-unless-clj identity form)
    (symbol? form) `(quote ~form)
    :else form))

(defmacro eval-clj
  [form]
  (quote-unless-clj form))

So then:

(eval-clj '(a (clj (+ 1 1)) b))

=> (a 2 b)

(let [x 42]
    (eval-clj '(a (clj x) b)))

=> (a 42 b)

  (let [x 1
        y 2
        z 3]
    (eval-clj '(a (clj (map inc [x y z])) b)))

=> (a (2 3 4) b)

Incidentally, you don’t have to quote, and you can quote mostly at random:

(eval-clj (a (clj (+ 1 1)) b)) => (a 2 b)

(eval-clj (a (clj (+ 1 1)) 'b)) => (a 2 b)

Some notes:

  • The calls to clojure.walk/walk, the way I call them, are just convenient ways to empty out a form and fill it with transformed items. Basically like map except it ensures the container remains the same as the original instead of always becoming a seq. See: walk - clojure.walk | ClojureDocs - Community-Powered Clojure Documentation and Examples
  • Overall, essentially all I’m doing is quoting things that need to be quoted when evaluated (implicitly) in the defmacro – lists and symbols – and leaving everything else unquoted. For the (clj...) and (quote...) forms I just basically hoist whatever the next item is.
  • I detect lists using seq? because the top-level form comes (or can come?) into the macro as a clojure.lang.Cons instead of a proper list. The seq? predicate detects it either way.
  • This eval-clj will descend into any data structure and find forms like (clj foo) and leave the form after clj unquoted - in other words, it doesn’t just work on lists, or things at the top level.

Update: I just noticed your mention that the DSL code would be available only at runtime, I updated my intro above as this obviously won’t help you in that case. See my next answer below for a possible solution.

6

Feels like this is just quasiquoting exposed. I think backtick supports this use case out of the box with its templating:

user=> (require '[backtick :as bt])
nil
user=> (let [x 1
             y 2
             z 3]
        (bt/template (a ~(map inc [x y z]) b)))
(a (2 3 4) b)

user=> (let [name "Sarah"]
          (bt/template (set-name ~name)))
(set-name "Sarah")
8

Sorry, I did not notice your requirement that your code is created at runtime. My prior solution will not be much help in that case. Here is another approach.

As @p-himik said, eval cannot get let bindings within forms. But those binding values are available at runtime and we can alter what gets passed to eval, replacing (in (clj...) forms) symbols with their bound values. (This is similar to an approach used in a different context by honeysql’s sql/formatv helper, which is where I first saw a pattern like this in use, as an alternative to the backtick/tilde syntax built in to Clojure, which gets awkward in certain circumstances.)

Using this approach here allows a solution to your problem.

Like this:


(require 'clojure.walk 'clojure.template)

(defn render-eval-clj
  [syms values form]
  (clojure.walk/prewalk (fn [form]
                          (if (and (seq? form) (= 'clj (first form)))
                            (eval (clojure.template/apply-template syms (second form) values))
                            form))
                        form))

(defmacro eval-clj
  [code]
  (let [syms (vec (keys &env))]
    `(render-eval-clj '~syms ~syms ~code)))

So then:

(let [code '(a (clj (+ 1 1)) b)]
  (eval-clj code))

=> (a 2 b)

(let [x 42
      code '(a (clj x) b x)]
  (eval-clj code))

=> (a 42 b x)

 (let [x 1
       y 2
       z 3
       code '(a (clj (map inc [x y z])) b x)]
  (eval-clj code))

=> (a (2 3 4) b x)

The only downside of this eval-clj is it relies on &env so only has access to local bindings. If you want to access global bindings you need to take as a vector argument the syms to make available to your (clj...) forms, same code as above plus:

(defmacro eval-clj-with
  [syms code]
  `(render-eval-clj '~syms ~syms ~code))

So then:

(in-ns 'foo)

(def bar 69)

(in-ns 'original-ns)

(let [x 70
      y 71]
  (eval-clj-with [foo/bar x y] '(a (clj (map inc [foo/bar x y])))))

=> (a (70 71 72))

Another downside is that, the way I wrote render-eval-clj, whatever comes out of your (clj...) form evaluation will itself be descended into, and any enclosed (but not top level) (clj...) forms resulting from the evaluation will themselves be evaluated. This is due to how clojure.walk/prewalk works. In the edge cases where this were to occur it may not be expected. You can avoid this by rewriting the function like so, it’s only slightly more verbose:

(defn render-eval-clj
  [syms values form]
  (cond
    (and (seq? form) (= 'clj (first form)))
    (eval (clojure.template/apply-template syms (second form) values))

    (coll? form)
    (clojure.walk/walk (partial render-eval-clj syms values) identity form)
    
    :else
    form))

Notes:

  • Symbols are not rendered unless they are inside (clj...) forms. (This could be trivially changed.)
  • eval-clj must be called with quoted code.

(Updates: Simplified code, removed reqiurement to pass in the symbols you want available to (clj...) form, swithed to using &env to get this automatically, added additional more correct render-eval-clj.)

9

Updated the above answer to no longer require a vector of syms be passed in to eval-clj.

10

To my surprise I found an easy solution.

(defmacro q [code]
  (cond (symbol? code)                            (list 'quote code)
        (number? code)                            code
        (string? code)                            code
        (keyword? code)                           code
        (vector? code)                            (apply vector (map #(list `q %) code))
        (map? code)                               (update-keys (update-vals code #(list `q %)) #(list `q %))
        (and (list? code) (= (first code) 'uq))   (second code)
        (list? code)                              (apply list 'list (map #(list `q %) code))))

(comment
  (q 4) 
  (q b)
  (q "abc")
  (q ())
  (q [])
  (q (a))
  (q (a b))
  (q ({a b}))
  (q {a b})
  (q [a b c])
  (q (a [b c d] e))
  (q (a (b c d) e)) 
  (q (a (uq (+ 1 1)) b))
  (let [x 'b] (q 4))
  (let [x 'b] (q (uq x)))
  (let [x 'b] (q (a (uq x))))
  (let [x 'b] (q ({a (uq x)})))
  (let [x 'b] (q {a (uq x)}))
  (let [x 'b] (q [a (uq x) c]))
  (let [x 'b] (q (a [(uq x) c d] e)))
  (let [x 'b] (q (a ((uq x) c d) e)))
  (let [x 'b] (q ({a (uq x) c d} e)))
  (let [x 'b] (q (uq (map identity ['a (q ((uq x) c))])))))

The macro q quotes something and uq is unquote. It seems to work.