I’ve used http-kit in the past to get the response as a string via the :body key in the response map. You can then parse that JSON string into a clojure map using some libraries and work from there in clojure.
One approach is to use a library like data.json or cheshire or plenty of other clojure JSON handling libraries to parse the JSON request into a clojure map. Then just walk the map to find the :objectNumbers keys. It looks like you just need to get the collection of art objects from the map, so if it we call it parsed-map, then
(def art-records (get parsed-map :artObjects))
should return a vector of maps. Then just process those to get the the keys you want and build up any references you need (or maybe just store the records in a map, e.g.
Now you can lookup all the information for each one (you could also use the :title as the key instead if that makes more sense) as a nested map of maps.
It looks like there’s already a key under the path [:links :self] like https://www.rijksmuseum.nl/api/nl/collection/SK-A-1505 that may be useful too. So you may not have to do any additional parsing, or maybe just use the URL they provide and modify it to suit your needs.
There are additional libraries and techniques (even from clojure.core) that can help with parsing very complicated nested data, but this case looks simple. I think these libraries or analogues are available in cljs as well, so you could probably have a browser-based SPA as well and maybe learn some additional tools later (like re-frame or reagent for ui stuff).
To translate the JSON to Clojure data, you can use Cheshire or some other wrapper of the Java “Jackson” library.
As for how to keep people from waiting, the universal solutions are (1) do as much as you can ahead-of-time, and (2) show annoying pop-ups with concealed x buttons that people enjoy hunting for.
You might displease the Museum if you slurped all 4000 object-number URLs in a hurry - or, worse yet, ten-at-a-time in different threads. It is polite to space out such work. Therefore, you might consider building your own digest of the Museum data at a leisurely pace. Then refer to it in “real time” when your website gets a visitor.
Rich Hickey famously said, “[Software] design is: taking things apart”. So: not just a background job, but an entirely separate program. That way, folks can enjoy your website while you tinker with the metadata-gathering algorithm, or you could easily shut it off for a little while if they sent an agent to speak with you about your bandwidth consumption.
By metadata, I mean information-about-the-art. You will not need to download the art itself. (I am supposing that your web page will put the Museum’s gif url into the <img…> tag so your visitors will get the actual art directly from the Museum’s webservice. If the Museum permits such a usage pattern, it brings the twin benefits of lower bandwidth usage for you, and true usage metrics for the Museum.)
Don’t over think it. Just start with the naive approach, it might already be fast. And if it isn’t, look into ways to make it faster. You could add a CDN to your site. You could add a small cache between you and the API, you could parallelize requests, etc. But you don’t have to think about those yet, because it’s too speculative, who knows which part is slow if any.