Clojure data into JSON and back with spec and Liberator

[otwieracz]

Hey!

I’ve ran into (obvious) issue when building JSON REST API with Liberator for my app. Main configuration is just nested structure of Clojure data (EDN) stored in file. Goal of the REST API is to expose fragments of this data to user (as Liberator resource), and update it with user-provided version via PUT. Easy peasy, right?

Nope. My configuration structure is not JSON! All (important) information about data type (set, hash-map, keyword, etc) is lost.

See this example:

(s/def ::name string?)
(s/def ::kind #{:dog :cat :mice})
(s/def ::friends (s/coll-of ::name :kind set?))
(s/def ::entity (s/keys :req-un [::name ::kind ::friends]))
(s/def ::entities (s/coll-of ::entity))

(def entities
  [{:name "Spike"
    :kind :dog
    :friends #{"Jerry"}}
   {:name "Tom"
    :kind :cat
    :friends #{"Jerry"}}
   {:name "Jerry"
    :kind :mice
    :friends #{"Tom" "Spike"}}])

(spec/valid? ::entities entities) ;; => true

(json/encode entities) ;; => "[{\"name\":\"Spike\",\"kind\":\"dog\",\"friends\":[\"Jerry\"]},{\"name\":\"Tom\",\"kind\":\"cat\",\"friends\":[\"Jerry\"]},{\"name\":\"Jerry\",\"kind\":\"mice\",\"friends\":[\"Tom\",\"Spike\"]}]"

(json/decode (json/encode entities)) ;; => ({"name" "Spike", "kind" "dog", "friends" ["Jerry"]} {"name" "Tom", "kind" "cat", "friends" ["Jerry"]} {"name" "Jerry", "kind" "mice", "friends" ["Tom" "Spike"]}) 

(spec/valid? ::entities (json/decode (json/encode entities))) ;; =>false

Information about all non-JSON data types is lost during conversion.

This seems like impossible case but wait - there’s spec with all the information about valid data types specified. If only I can somehow coerce into this spec…

I don’t know if Liberator in original application is important - it’s doing JSON conversion implicitly, but in my opinion eventual solution will be independent of Liberator - but mentioning it, just in case.

Do you have any ideas how to deal with it?

Slawek

[mdiin]

As noted in this StackOverflow answer, which also links to this mailing list thread: Clojure.spec is not intended for coercion. The approach recommended by the core team is to use plain functions to coerce data before validating with spec.

We have not transitioned from schema to clojure.spec yet, but I imagine that if/when the time comes we will keep schema as the library in use for coercing input at the boundaries of our APIs, and just use clojure.spec for describing the shape of data at key points in the code base. This answer by @didibus is what I would use as a reference for which functions to spec when the time comes.

[ikitommi]

For now, you can:

• Use Schema, it’s does this well and is well maintained
• try tools like spec-tools or spec-coerce

The latter uses s/form to parse (most of the) specs at coercion time, the first uses combination of spec parsing and conforming, which is concidered a bad practise.

If we could walk over the specs, we could have a fast real spec-coercion as a 3rd party lib, just like with Schema, without hacks or runtime parsing of spec forms.

Wrote a post about Spec Transformers a while back, using spec-tools.

[didibus]

Yes, I really hope we end up with Specs we can walk over, as well as attach extra meta info to them also.

I’d love to be able to use Spec to generate APIs and the like.