Continously calculate and deliver data it with ring

[habruening]

I have these few lines of code that creates a lazy list of numbers and sends them out over http with ring.

(defn lines
  ([] (lines 1))
  ([n] (lazy-seq (cons (str n "\n") (do (Thread/sleep 1000)
                                        (lines (inc n)))))))

(defn handler [request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body (take 5 (lines))})

(def server
  (ring.adapter.jetty/run-jetty handler {:port 3000
                      :join? false}))

The behaviour is that the request is blocked for 5 seconds and then all numbers are sent out. But I want to achieve that every second one number is sent out. I don’t ask here for a solution, but for a hint, what I should learn, to achieve this.

At the moment I have no idea how to start. Are lazy sequences made for this purpose, at all? I tried to investigate if perhaps Java IO Stream could be used, because Ring responses would accept them. But this does not feel like Clojure. Would this be the right approach? Creating an output stream in the function lines and passing this as an input stream to the response?

I also found that library. But with my limited experience I don’t see if this library addresses my problem. Should I first learn how Clojure Async works?

I suppose I need something that enables parallel execution. But interestingly, when I open 5 browser instances simultaneously, all 5 requests already work in parallel. But I don’t manage to use this sort of parallelism for my goal. It somehow makes sense to me that it does not work, because things are immutable. So I cannot simply create a calculation that magically works parallel to everything else and that every time we ask it for a number delivers something else.

But how can I achieve something that produces something over the time?

[Harold]

What you’re trying to accomplish isn’t perfectly clear, but since it seems that you’re in an exploratory mode, perhaps look into Server Sent Events (Server-sent events - Web APIs | MDN), which is something we’ve used in the past in similar (but probably not identical) cases.

[joinr]

I think generating the response body (currently) has to realize the sequence from take. So waiting n seconds (depending on the items you draw) is going to be the bottleneck.

Without something to stream output in the response, this is going to continue to be an issue.

[seabass]

This comment is not related to Clojure specifically, but are you sure you want to do it this way? HTTP isn’t really supposed to be used synchronously like this: clients won’t necessarily like that your server keeps them waiting for the end of the file, and there’s no way of restarting it if the connection is broken. HTTP ‘chunked transfer coding’ is more suitable, but has a lot of overhead if your payload is literally just a few digits each time. The best way of doing this would probably be WebRTC, which is a proper streaming protocol initiated by HTTP, and which also supports NAT hole punching.

[thheller]

You can achieve something that produces things over time. Should you? No, absolutely not. SSE is a solution. Websockets are even better than that. WebRTC is bonkers and should not be a consideration.

It is totally possible to send indiviual chunks over the write, even without the (deprecated) Transfer-Encoding: chunked, and you can fully control what the server does. You however have no real control over what the Browser does with it. Most browsers support some sort of streaming HTML, but whether or not something gets rendered immediately is not something you have any control over. Which is why Websockets are a much better fit and even allow two way communication. Regular HTTP requests should really just complete ASAP.

Here is a solution if you just want to mess arround. This blocks the entire ring thread for however long this is running, and there are only very few of those threads. So never even think about running this kind of code in any kind of production setup with more than 1 user.

But I have to admit that I have done similar things in the past, although Websockets didn’t exist back then. So I would not consider doing something like this again.

(ns main
  (:require
    [ring.core.protocols :as rp]
    [ring.adapter.jetty]))

(defn handler [request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body
   (reify rp/StreamableResponseBody
     (write-body-to-stream [_ response out]
       (loop [i 5]
         (when (pos? i)
           (.print out (str "hello:" i))
           (.flush out)
           (Thread/sleep 1000)
           (recur (dec i))
           ))
       (.print out "done")
       ;; must call close, otherwise connection stays open until timeout
       (.close out)))})

(comment

  (def server
    (ring.adapter.jetty/run-jetty handler {:port 3000
                                           :join? false}))

  (.stop server))