Spec: entity id list and entity consistency?

Hi, I was wondering if anyone knows the answer or could point me at a resource that would help me solve this problem.

I want to generate a list of entity-ids and then a list of entities, the catch is I want each entity in the list to be generated with an id that corresponds to a list from the entity id.

(s/def ::id (s/with-gen string? #(s/gen #{"a" "b" "c" "d"})))
(s/def ::entity-ids (s/coll-of ::id :count 2 :distinct true))
(s/def ::entity (s/keys :req-un [::id]))
(s/def ::entities (s/coll-of ::entity :count 4 :distinct true))
(s/def ::manifest (s/keys :req-un [::entity-ids]))
(s/def ::manifests (s/coll-of ::manifest :count 2 :distinct true))
(s/def ::foo (s/keys :req-un [::manifests ::entities]))

(gen/generate (s/gen ::foo))

;; output

{:manifests [{:entity-ids ["d" "a"]}
             {:entity-ids ["a" "b"]}], 
;; c is in neither of the manifests entity lists
 :entities  [{:id "a"}
             {:id "d"}
             {:id "c"}
             {:id "b"}]}

;; But what I want is below.
;; Each entity has an id that maps to at least one of the entity-id lists

{:manifests [{:entity-ids ["a" "b"]}
             {:entity-ids ["c" "d"]}],
 :entities  [{:id "a"}
             {:id "d"}
             {:id "c"}
             {:id "b"}]}

If I haven’t made the problem clear enough, please let me know and I’ll try and elaborate. Thanks in advance! :smiley:

Hello!

You might want a custom generator for that.

Note that your ::foo spec is valid even when the ID’s aren’t consistent (as you’re noting). You could use a predicate for that. Something like

(defn consistent-foo? [foo]
  ;; ...
  )

(s/def ::foo
  (s/and consistent-foo?
         (s/keys :req-un [::manifests ::entities])))

Teodor

1 Like

That’s the part that I was missing! I completely forgot that s/and uses the first predicate/spec as a generator and those after it can narrow down the results. It makes perfect sense, like you said my spec for foo wasn’t capturing the constraint that I wanted, so it obviously wouldn’t generate the correct data.

Here’s what I ended up with if anyone is interested:

(s/def ::id (s/with-gen string? #(s/gen #{"a" "b" "c" "d"})))
(s/def ::entity-ids (s/coll-of ::id :count 2 :distinct true))
(s/def ::entity (s/keys :req-un [::id]))
(s/def ::entities (s/coll-of ::entity :count 4 :distinct true))
(s/def ::manifest (s/keys :req-un [::entity-ids]))
(s/def ::manifests (s/coll-of ::manifest :count 2 :distinct true))

(defn consistent-foo? [{:keys [manifests entities]}]
  (= (set (mapcat :entity-ids manifests))
     (set (map :id entities))))

(s/def ::foo
  (s/and (s/keys :req-un [::manifests ::entities])
         consistent-foo?))

(gen/generate (s/gen ::foo))

;; output

{:manifests [{:entity-ids ["b" "c"]}
             {:entity-ids ["d" "a"]}],
 :entities  [{:id "b"}
             {:id "a"}
             {:id "d"}
             {:id "c"}]}

Thanks Teodor! :smiley:

1 Like

Glad to hear it helped!

This is useful stuff… I have a similar (but a bit larger scope) problem, and this looks like a good starting point for solving it. Thanks guys!

Posting this in case its useful to anyone.

I started running into issues where spec was unable to generate large consistent sets in 100 tries. The solution I found was to use a custom generator with single item set that guaranteed that the manifest and the items would have the same keys. I don’t feel like this is a particularly elegant solution but it does work.

The data format is slightly different to that in my initial question.

{:manifest {:item-order ["b" "c"]},
 :items  [{:id "b" :type "food"}
          {:id "c" :type "drink"}]

Here’s my approach.

(require '[clojure.spec.alpha :as s])
(require '[clojure.spec.gen.alpha :as gen])

(defn large-shuffled-vec-of-ids []
  (->> (range 1000)
       (map (partial str "xid"))
       shuffle
       vec))

(s/def ::id string?)
(s/def ::item-order
  (s/with-gen
    (s/coll-of ::id :distinct true)
    #(s/gen #{(large-shuffled-vec-of-ids)})))
(s/def ::manifest (s/keys :req-un [::item-order]))
(s/def ::type #{"food" "drink"})
(s/def ::item (s/keys :req-un [::id ::type]))

(s/def ::items
  (s/with-gen
    (s/coll-of ::item :distinct true)
    #(s/gen #{(let [ids (large-shuffled-vec-of-ids)]
                (map (fn [item id] (merge item {:id id}))
                     (apply concat (s/exercise ::item (count ids)))
                     ids))})))
                     
(defn consistent-ids? [{:keys [manifest items]}]
  (= (set (manifest :item-order))
     (set (map :id items))))
     
(s/def ::items-with-manifest
  (s/and
   (s/keys :req-un [::manifest ::items])
   consistent-ids?))
   
(comment 
   (gen/generate (s/gen ::items-with-manifest)))   

Is there a better approach that I’m just missing?

Hey @andersmurphy,

Just read the PurelyFunctional.tv newsletter, (read issue 330 in full), which mentioned this. You’ve got your solution, but if you’re interested in more context, the newsletter might give you that.

@ericnormand provides an example for E-mail generation. Snipped from the newsletter (hope that’s okay):

(def email-re 
#"(([^<>()\[\]\.,;:\[email protected]\"]+(\.[^<>()\[\]\.,;:\[email protected]\"]+)*)|(\".+\"))@(([^<>()\[\]\.,;:\[email protected]\"]+\.)+[^<>()\[\]\.,;:\[email protected]\"]{2,})")

(def email-char-re #"[^<>()\[\]\.,;:\[email protected]\"]")

(def gen-email-char
  (gen/such-that #(re-matches email-char-re (str %)) gen/char))

(def gen-email-char-string
  (gen/fmap #(apply str %)
            (gen/not-empty (gen/vector gen-email-char))))

(def gen-email-name
  (gen/fmap #(str/join "." %)
            (gen/not-empty (gen/vector gen-email-char-string))))

(def gen-email-domain
  (gen/fmap (fn [[domain-segs tld1 tld2]]
              (str domain-segs "." tld1 tld2))
            (gen/tuple gen-email-name
                       gen-email-char-string
                       gen-email-char-string)))

(def gen-email
  (gen/fmap (fn [[username domain]]
              (str username \@ domain))
            (gen/tuple gen-email-name gen-email-domain)))

(gen/sample gen-email)
;; => ("‡@¥.ú .²e" "Ø[email protected]ñ.NÎ" "¿G.Ø@ñwßÃ..ëxÂ!" "š¨.Ðâë.¼/.Âù@òÉ.Â.…ë.‚ù" 
;;     "vu.BÖ¯¢.ƒÍ@þÛü..´Ê!6" "Â.L¹¹èf.Zô.æ@\b*î.0¡6.½.ƒŒ¶" 
;;     "ÏÂÆ{É.ûÈÉu.âà[email protected]€WÂ.µR*A.Hcõ&E.PE$.òIÛOÚäû" 
;;     "ž&9ÅÂÇ.Où¸.r¾.Þa&[email protected]#mõê.cӝ\\æ÷" 
;;     "Yù.íï®±|y.ÐNFÕÊ!š.TŒÔéã.ŒP.ipÂðêc}.PŒ@^P.I˸j¤ñ³.óZÀËa.ú}#zvá.Šm.^}öSFjØ.bžÀÊEšcÖ¡" 
;;     "æ}¯.sÇ.F4.Ý*fÝBº.V.ø.L–¥hôít².³ÒbžÕ@wHföª†«.â·cÔq¬.Yí-§ª.cÞÇ”\bñĽö.Æ×`’.í Oâ.ƒ³«˜Œ.šØ$¨ã§8ÿú/•")

1 Like

Awesome thanks for sharing! (signed up to the newsletter too) :slight_smile:

1 Like