How can I do this the best in clojure

Hello,

Im thinkig of trying my first web app in clojure and think of a gallery of paintings of the Rijksmuseum.

First I have to use this url: https://www.rijksmuseum.nl/api/nl/collection?key=14OGzuak&format=json&type=schilderij&toppieces=True

of that I need to filter out the objectNumbers.

then I need those objectNumbers to fetch the url of this url of the api : ‘https://www.rijksmuseum.nl/api/nl/collection/’ , objectNumber
, ‘?key=14OGzuak&format=json’.

and filter out the right url which is hidden under the z6, z5 or z4 key.

How can I do this so the user does not have to wait too long to see the paintings ?

I’ve used http-kit in the past to get the response as a string via the :body key in the response map. You can then parse that JSON string into a clojure map using some libraries and work from there in clojure.

One approach is to use a library like data.json or cheshire or plenty of other clojure JSON handling libraries to parse the JSON request into a clojure map. Then just walk the map to find the :objectNumbers keys. It looks like you just need to get the collection of art objects from the map, so if it we call it parsed-map, then

(def art-records (get parsed-map :artObjects))

should return a vector of maps. Then just process those to get the the keys you want and build up any references you need (or maybe just store the records in a map, e.g.

(reduce (fn [acc r] (assoc acc (get r :objectNumber) r)) {} art-records)

Now you can lookup all the information for each one (you could also use the :title as the key instead if that makes more sense) as a nested map of maps.

It looks like there’s already a key under the path [:links :self] like https://www.rijksmuseum.nl/api/nl/collection/SK-A-1505 that may be useful too. So you may not have to do any additional parsing, or maybe just use the URL they provide and modify it to suit your needs.

There are additional libraries and techniques (even from clojure.core) that can help with parsing very complicated nested data, but this case looks simple. I think these libraries or analogues are available in cljs as well, so you could probably have a browser-based SPA as well and maybe learn some additional tools later (like re-frame or reagent for ui stuff).

Getting stuff from a web URL does not require any power tools:

(slurp "https://www.rijksmuseum.nl/api/nl/collection?key=14OGzuak&format=json&type=schilderij&toppieces=True")

To translate the JSON to Clojure data, you can use Cheshire or some other wrapper of the Java “Jackson” library.

As for how to keep people from waiting, the universal solutions are (1) do as much as you can ahead-of-time, and (2) show annoying pop-ups with concealed x buttons that people enjoy hunting for.

You might displease the Museum if you slurped all 4000 object-number URLs in a hurry - or, worse yet, ten-at-a-time in different threads. It is polite to space out such work. Therefore, you might consider building your own digest of the Museum data at a leisurely pace. Then refer to it in “real time” when your website gets a visitor.

I was more thinking about 10 at the time for a page and when a visitor wants the next page ask again.
But Iif I understand you that is not handy.

I tried to place all the data in a database with ruby but that took me 3 hours of time to fetch all the data.
So I wonder if there was a better way

So you suggest to use a database and let a back-ground job do the fetching ?

Rich Hickey famously said, “[Software] design is: taking things apart”. So: not just a background job, but an entirely separate program. That way, folks can enjoy your website while you tinker with the metadata-gathering algorithm, or you could easily shut it off for a little while if they sent an agent to speak with you about your bandwidth consumption.

oke,

so still store the data in a database.
Sorry to ask so many questions but I try to see how I could do things.

and what do you mean wth meta-data ?

By metadata, I mean information-about-the-art. You will not need to download the art itself. (I am supposing that your web page will put the Museum’s gif url into the <img…> tag so your visitors will get the actual art directly from the Museum’s webservice. If the Museum permits such a usage pattern, it brings the twin benefits of lower bandwidth usage for you, and true usage metrics for the Museum.)

yep, the api gives a url where the image is found.
That is why I asked how I can do it the best in clojure

Ask the objectnumbers and then for 10 numbers ask the image-url and or display it or store is somehow

Don’t over think it. Just start with the naive approach, it might already be fast. And if it isn’t, look into ways to make it faster. You could add a CDN to your site. You could add a small cache between you and the API, you could parallelize requests, etc. But you don’t have to think about those yet, because it’s too speculative, who knows which part is slow if any.

so first make the code to deal with the objectNumbers and then make code to fetch the urls for the images and use them after each other.

Is that what you mean

Exactly. Just try it like that, and if you find it too slow, than deep dive what part is taking too long and brain storm ideas to speed those up.