Need help to resolve symbols programmatically

Community Center Beginners
21

That’s why Clojure has the backquote, which resolves symbols as it reads the list.

`(or (satisfies list?) 
     (and Integer (not (satisfies odd?))))

That said, if you were using satisfies as a magic symbol and not as a real pointer to a satisfies function, they’d need to escape that from being resolved.

Like in this scenario, is the satisfies supposed to point to a var from your.lib ? Or the user namespace? Or neither, and you interpret that symbol as is in your code?

(ns my.ns
  (:require [your.lib :as l]))

(defn satisfies [a b] (println a b))

(l/define
`(or (satisfies list?) 
     (and Integer (not (satisfies odd?)))))

If it refers to a var in your.lib, the user should do:

(l/define
`(or (l/satisfies list?) 
     (and Integer (not (l/satisfies odd?)))))

So it gets resolved to the right thing when backquoted. If it was supposed to refer to the var they defined, then you’d keep it how it was. But if you want it to be interpreted they’d need to do:

(l/define
`(or (~'satisfies list?) 
     (and Integer (not (~'satisfies odd?)))))

So I’ve seen people prefer keywords in those cases:

(l/define
`(or (:satisfies list?) 
     (and Integer (not (:satisfies odd?)))))

Or I’ve seen people define in their lib a var of the symbol that returns the unqualified symbol or keyword like:

(ns your.lib)

(def satisfies 'satisfies)

(ns my.ns
  (:require [your.lib :as l :refer [satisfies]]))

(l/define
`(or (satisfies list?) 
     (and Integer (not (satisfies odd?)))))

In my opinion this, or having your code resolve things at the place it is given would be best.

The thing is, when the user provides unqualified symbols, that’s just not the real pointer. It’s like a shortname. So if you did that, you could bank on the fact their won’t be conflict, or do what CL does, and do a search but if you find conflicts throw an error and then ask them to explicitly shadow the one they want. And similarly you could let them define nicknames too.

(ns my.ns
  (:require [your.lib :as l])

(l/set-types!
  {'list? `list?
   'odd `odd?
   'Integer `Integer})

(l/define
`(or (satisfies list?) 
     (and Integer (not (satisfies odd?)))))

As an aside, I’m still confused how you solved it in Common-lisp?

What happens if someone gives you a shadowing-import symbol? Or if they use a package local nickname?

22

In common lisp quote and backquote behave the same except for , and ,@. This means '(and) is the same as `(and) … The elevates most of the problem.

In clojure, backquote is very difficult to use on non-code. Common lisp backquote does interpolation, bit not symbol resolution. The parser does symbol resolution.

23

satisfies is just the symbol satisfies, no namespace, no var. Same for =, member, and, or, not, xor,…

24

Keywords have a different meaning.

25

I meant how you handle shadowing-import and local nicknames in CL? Since those are user contextual the same as in Clojure they don’t properly disambiguate.

You mean in your DSL?

Ok, well it’s up to you to make a choice here. I can’t think of any other way of doing it.

Personally, I would go for either resolving in place, or having users give you fully qualified symbols.

Resolving in-place, something like:

(defn define
  [lst]
  ;; Here you ignore your DSLs symbols like or and
  ;; satisfies, but call resolve on *ns* for the others
  )

(define
  '(or (satisfies list?) 
     (and Integer (not (satisfies odd?)))))

Or having the user provide fully qualified:

(ns your.lib)

;; In your lib you define all symbols for your DSL
(def or 'or)
(def and 'and)
(def satisfies 'satisfies)

;; And when you need to process the provided DSL, you resolve
;; all symbols within the context of your namespace

(defn define
  [lst]
  ;; Here you'd resolve it all within your namespace
  ;; so that your.lib/or becomes 'or
  )

;; In the user namespace they refer to those or alias them at their discretion
(ns user
  (:require [your.lib :as lib :refer [satisfies]]))

(lib/define
  `(lib/or (satisfies list?) Integer))
26

No, it’s not a problem. The CL reader always reads symbols as a function of the current package. The current package has import lists of other packages, and the symbol resolution is well defined.

In the case that you want to import symbols from other packages and deal with conflicts, there are ways of doing this, and if you want to import symbols from another package but not some from that package, that’s possible as well.

However, give a package in a current state, the read function reads into that package. If there are conflicts, conditions are raised, and there may be condition handlers in place to handle them. For example if read encounters text such as aaa:bbb or aaa::ccc and package aaa does not exist, or does not export the symbols bbb or does not contain the symbol ccc, then conditions are raised.

27

I think I will indeed have to write a resolution engine which understands my semantics.
It won’t be so difficult to find all places where a class name or function name is expected, and replace these with symbols from a lookup table, defaulting to clojure.core or java class if not found in the lookup table.

Unfortunately, the clojure backquote does resolution in a way which works somewhat for code, but doesn’t work so well for data. And I don’t think it is possible for the closure programmer to write something like backquote but with slightly different behavior. In common lisp it would indeed be possible, because programs can define reader macros.

As I understand, reader macros are not supported in clojure.

28

@didibus, thanks for all the advise.

29

Yes my DSL already has a meaning for keywords such as :and :or :not …

30

as I understand your suggestion I’d need to ~' the and, or and not as well, because

'(and x y z)

gives a different and than does

`(and x y z)

Thus…

(l/define
`(~'or (~'satisfies list?) 
     (~'and Integer (~'not (~'satisfies odd?)))))
31

which satisfies function are you referring to? I don’t find any such function in clojure.core. I find satisfies?, but not satisfies. And, yes, I’m using satisfies just as the symbol satisfies, not as a var and not as a namespaced symbol.

32

Have a look at https://github.com/brandonbloom/backtick, the template macro might address some of the issues you are facing.

1 Like
33

That’s what confuses me. You said that your library parses the DSL in a different namespace then where it is provided? That would mean the CL reader would read the DSL in the context of the package where it is defined. In that package maybe I have X → aaa:X.

Now you are in a different package, and you say that you will now make use of the DSL, so now you see the symbol X in your DSL, but you are in a different package, so how do you know it is aaa:X given the symbol X and not bbb:X ?

It seems to me this wouldn’t work, unless the user gave you an unambiguous symbol like aaa::x

I think you can do it in a normal macro. Don’t think this needs to be a reader macro to work. Just selectively call resolve only on symbols that you know you want resolved.

You can define a data reader macro, but it can’t get access to the text directly, it will be given the read syntax, but I believe it will execute before other macros do.

I might have suggested too many things now I’m not sure which one this was. If it was the one with set-types I meant to use the quote for that one, not backquote. And then in your library you’d use the provided mappings in set-types to resolve the real vars and classes.

For the other one, I’m saying if you have the user use backquote, but you provide Vars from your namespace that contain the unquoted symbol, it can be a nice syntax. Though the user would need to either refer or alias your symbols like:

(ns user
  (:require [your.lib :as lib :refer [or and not satisfies]]))

(lib/define
  `(or (satisfies list?) 
     (and Integer (not (satisfies odd?)))))

So on this case you’d get:

(your.lib/or (your.lib/satisfies clojure.core/list?) 
     (your.lib/and java.lang.Integer (your.lib/not (your.lib/satisfies clojure.core/odd?)))))

But, if you define your.lib/or and the other inside your lib to be:

(ns your.lib)
(def or 'or)
(def and 'and)
...

Then before applying your DSL, you can resolve all symbols, and you’d get everything back the way I think you want it.

Also, I just thought of another option, you could capture the value of ns from when they gave you the DSL, and when you use the DSL you can resolve with ns-resolve and the namespace that you had captured from earlier. Assuming that namespace isn’t in some other app, then it would give you everything you want.

34

In common lisp the symbols are assigned packages at read time. So at parse time I don’t have to make any decision about packages. They’re already assigned. If a reader has a symbol not (for example) in his package which shadows CL:not, then yes it will be read as my-package:not, and will not be the same as CL:not — thus my DLS interpreter (which in no way depends on *package*) will not understand it. This is what the CL user expects. I.e., if the application programmer shadows low-level names (such as and, not, if car, cdr), he expects to have lots of grief.

If the DSL interpreter depended on *package*, then a program would stop working when the user puts his repl in a different package, which as I see it is a dangerous problem of run-time calls to resolve in clojure.

35

But in that case would the programmer still be able to use and, or, and not within his program logic for their normal clojure.core semantics? If not, that would be a disaster and a hostile programming environment. I don’t want to change the semantics of built-in functions/variables in normal program code, I only want their semantics changed within the DSL.

37

@mjmeintjes, wow that looks really nice. I’ll look into it. it appears to be a backtick for data structure interpolation that is not plagued with the awkward problem of namespace resolution.

It’s really interesting to see that someone else has had the same problem as me, and come up with an elegant solution.

38

They have many options that’s up to them to choose from.

They can use an alias:

(ns user
  (:require [your.lib :as yl]))

(yl/define
  `(yl/or (yl/satisfies list?) 
     (yl/and Integer (yl/not (yl/satisfies odd?)))))

They can rename the vars:

(ns user
  (:require [your.lib :as yl
                      :refer [or and not satisfies]
                      :rename {or oor, and aand, not nnot}))

(yl/define
  `(oor (satisfies list?) 
     (aand Integer (nnot (satisfies odd?)))))

They can accept to have it shadowed in certain namespaces:

(ns user
  (:require [your.lib :as yl
                      :refer [or and not satisfies]))
;; If as a user I know I'm not going to use them anywhere else
;; in the current namespace
(yl/define
  `(or (satisfies list?) 
     (and Integer (not (satisfies odd?)))))

;; And if I really need too, I can use fully qualified
;; symbols to refer to the core ones:
(clojure.core/or 1 nil 2)

Finally they can choose to alias or rename the core symbols if they prefer:

(ns user
  (:require [your.lib :as yl
                      :refer [or and not satisfies]
            [clojure.core :as c :rename {or cor}]))

(yl/define
  `(or (satisfies list?) 
     (and Integer (not (satisfies odd?)))))

;; Now I can use the renamed core symbols or the core alias to refer to them
(cor 1 nil 2)
(c/and true true)

Now my guess is if you go with this approach, almost all users will just choose to alias your library and use the alias with all symbols from it.

1 Like
39

I didn’t quite understand the semantics of your suggested yl/define function.

40

I am still not seeing how this is an actual problem yet in practice. The original issue was resolving symbols defining classes at runtime based on unqualified user code. I think there were a couple of schemes discussed. The push seems to be toward replicating CL’s behavior, which means qualifying at readtime. Without hacking the reader (which we actually can and implement reader macros in a non-portable fashion, which is undesireable), we can mess with reader literals to define a qualified reader pretty easily.

data_readers.clj

{qualified         dsl.core/qualified-read-any}

dsl/core.clj

(ns dsl.core)

(defmulti qualified-read read-type)

(defmethod qualified-read :symbol [x]
  (symbol (name (ns-name *ns*)) (name x)))

(defmethod qualified-read :map [coll]
  (reduce-kv (fn [acc k v]
               (assoc acc (qualified-read k) (qualified-read v)))
               (empty coll) coll))

(defmethod qualified-read :collection [coll]
  (into (empty coll) (map qualified-read coll)))

(defmethod qualified-read :literal [x] x)

(defn qualified-read-any [[x]]
  (qualified-read x))

user> '#qualified[(or (satisfies list?) (and Integer (not (satisfies odd?))))]

((((user/odd? user/satisfies) user/not) user/Integer user/and)
 (user/list? user/satisfies)
 user/or)

Not sure what this buys you over ` though (it’s not clear to me what was problematic about it in your use case); but it’s another avenue toward packing the namespace in with the quoted stuff. You are free to do all the parsing you want at readtime just by messing with the multimethods. You could trivially scan through the symbols that are known to the dsl and do custom processing or replacement, etc.

41

I’ve also thought about just having my DSL understand clojure.core/and the same as non-namespaced and. There are only a handful of such symbols anyway.