Interesting idiosyncrasy of the compiler

Questions & Help Troubleshooting
1

The following code complies without errors or warnings, but it doesn’t behave as I expected. The problem is that the function f is forward declared and then the function g is compiled acknowledging that declaration. Later f is defined as a dynamic variable and no error/warning is issued.

When function h rebinds f using binding the call to f within the function g is not effected by this rebinding.

The fix is to change the declaration to (declare :dynamic f).

Of course in this very small test case, it might be obvious what’s wrong. But my original problem was in a large file where the declaration, definition, and use sites were very far away from each other.

Shouldn’t the compiler warn me if I define a dynamic variable which was declared to be non-dynamic?

(ns jimka-test
  (:require [clojure.pprint :refer [cl-format]]))

(declare f)

(defn g []
  (map f '(1 2 3)))

(def ^:dynamic f (fn [x] (* x x)))

(defn h []
  (assert (= (g) '(1 4 9)))
  (binding [f (fn [x] (+ x x))]
    (assert (= (g) '(2 4 6))
            (cl-format false "(g) returned ~A" (g)))))

(h)
2

Interesting. You created a closure over a forward declaration of f in g, then redefined (remapped) f as a dynvar where g is unaware based on its original inspection of the metadata from f’s forward declaration.

Shouldn’t the compiler warn me if I define a dynamic variable which was declared to be non-dynamic?

Sounds reasonable. Curious if there is patch for this already pending in jira. Might be such a corner case (forward declare for me is typically a code smell and I try to re-organize if possible), so I’m curious if anyone else has actually hit it in the wild.

I would submit this example to ask.clojure.org since that is the official community repository for flagging stuff like this (and getting into JIRA without having a dev account; basically Alex Miller is using it for issues and general stack oveflow replacement).

3

It could but it would need to do a whole-program analysis. If I understand correctly, the behavior you’re seeing is due to the fact that Clojure programs are read and interned as if at a REPL; using a var which is interned as dynamic at the time of being read emit different code than if they are not marked as dynamic at the time of reading.

Example:

(ns jimka-test
  (:require [clojure.pprint :refer [cl-format]]))

(declare f)

(defn g []
  (map f '(1 2 3)))

(def ^:dynamic f (fn [x] (* x x)))

(defn g' []
  (map f '(1 2 3)))

(defn h [g]
  (assert (= (g) '(1 4 9)))
  (binding [f (fn [x] (+ x x))]
    (assert (= (g) '(2 4 6))
            (cl-format false "(g) returned ~A" (g)))))

(h g)
;; => Assert failed: (g) returned (1 4 9)
;;      (= (g) (quote (2 4 6)))

(h g')
;; => nil
4

Until now I have resisted wording my functions to avoid declarations. I prefer to put related functions together, so that features are implemented by a set of consecutive functions. This often does not match the dependency topology.

5

Looks like using the var directly (e.g. via #’) works:

(defn g []
  (map #'f '(1 2 3)))
1 Like
6

I don’t see why it would need whole-program analysis. As I understand, maybe I’m thinking too simplistic, declare tells the compiler (among other things) that the variable IS NOT dynamic. Then the (def ^:dynamic ..) says it is dynamic. There could be a warning here saying WARNING variable changed from lexical to dynamic.

7

I don’t know what’s wrong here, this seems to meet my intuition.

You declared a non-dynamic var f and closed over it. It then behaves in a non-dynamic way.

What happens if you use the dynamic f later in? Is it not dynamic?

8

A warning might be good when changing a var to dynamic, yeah. It’s a corner case I’m guessing few people have met, since there are a small number of instances using dynamic vars makes sense in application code, which explains to me why there are some sharper corners on it.

1 Like
9

moving the function lower in the file changes the functions behavior. That should be surprising, right? After a simple move of a function, everything still behaves correctly, as the function is still defined in VM. but after a restart, it fails.

10

Also keep in mind declare is not a compiler declaration, it is simply a macro implemented as:

(defmacro declare
  "defs the supplied var names with no bindings, useful for making forward declarations."
  {:added "1.0"}
  [& names] `(do ~@(map #(list 'def (vary-meta % assoc :declared true)) names)))

All it does is:

(def ^:declared foo)
1 Like
11

OK, done. asked on ask.clojure

12

Look at this:

(declare foo)

(def first-foo #'foo)

(def ^:dynamic foo
  (fn[] 1))

(println "identical: "
  (identical? #'foo first-foo))

;> identical: true

def will change what the Var points too if the binding already exist.

declare will create a Var and bind it in the current namespace.

When you call declare it creates a non-dynamic Var and binds it to f in the current namespace, because under the hood it just calls def.

When you call def again, it will see that f is already bound, so it will not create a new Var and it will not rebind f, all it will do is change the root value of the existing Var bound to f, which is the one created by declare which is a non-dynamic Var.

What that means is that in Clojure bindings are immutable, once you’ve created a Var and bound it to a symbol in that namespace it can never be changed to point to anything else. When you re-def something it’ll update the existing var-root, you cannot change the type of Var itself, which is why you can change an existing binding from a non-dynamic Var to a dynamic Var.

What you can do is put ^:dynamic on declare as well, and that way it’ll create a dynamic Var.

(declare ^:dynamic f)

But remark also that when you def, while if it is already bound to a Var it won’t rebind to a new Var, it will modify the metadata and the root value of the existing Var.

There is a compiler check in here for trying to def without ^:dynamic on a dynamic Var. I guess there could be one added.for the other way around ideally to give an error.

Edit:

Hum, maybe this is wrong, your second example seems to contradict what I said. I would expect f to simply never be dynamic, even if you close on it after.

There might be something about dynamic Var that I don’t understand, which seems to maybe relate to their metadata, and I’m not sure what happens to that on the closure.

13

I think what really happened was that I used declare to be able to create a function referencing a perfectly normal variable. After several months I started refactoring the code because it was too slow (python and scala versions of the same code were much faster). One refactoring was to memoize certain functions, but only in certain dynamic contexts where certain groups of functions were highly mutually recursive. The memoization served to prune many exponentially explosive tree descents. To do this I changed the definition of a static function to a dynamic variable holding a memoized version of a static function, and used binding to rebind the variable to a fresh memoization of the same static function.

This worked very well. The code was much faster, and all my 1000s of tests still passed except for 3 tests which were very curious. Step-by-step debugging of these 3 tests eventually led me to ask the question on clojurians, when the answer became evident.

I created the overly-simplified code in this test case to demonstrate the principle.

14

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.