Hi all recently I have seen some talks from Rich where he encourages the usage of queue for reducing complexity. So I was searching for examples in the wild used in a clojure projects and hints in this direction for learning more about this. Tia
Was he talking about core.async?
Core.async channels are concurrent blocking queues. You put things in them, and you take things from them, on a first in first out basis.
A producer can put while a consumer takes.
If the producer is producing too fast, it will block and wait before putting, say if the channel is full.
If the consumer is consuming too fast, it will block and wait, say the channel is empty, so it will wait for something to be put into it.
Producers can be consumers and vice versa. I which case they are normally called workers.
Workers can be each in their own thread. And they can use channel to send data between each other across threads, in a synchronized manner. In which case, the OS decided preemptively how much work each worker is allowed to do until they must yield to another worker on another thread.
Workers can also all be in the same thread, but each worker can cooperatively choose to at some fixed point in their process interpret their work and yield to another worker on the same thread. Again, worker can use channels to send data between each other.
Core.async supports both mode of operations.
You can also choose to use Java’s executor service. And they often make use of ThreadPools, and concurrent queues to similarly schedule work over threads for n numbers of workers which can optionally communicate through the queue.
Oh, and since you linked to that brave and true page. You can think of promise as a one element, one time use, blocking queue. And future as a one element, one time use, blocking queue that also spawn a producer on a background thread which will produce one and exactly one value.
3 Likes
Thx. Afaik he was referring to the way we as programmer can untangle 2 objects or functions using queue for passing messages, instead of let B depends directly from A(I put objects word because I think this pattern concern also objects A and objects B in other Lang). It is definitely something that I didnt not have the chance to see a lot around in codebases.
Do you have any github projects links? I was actually searching for any projects using queues in the wild
I don’t really. I searched quickly and found this though https://github.com/tonsky/datascript-chat
1 Like
Thx!
If anyone has other Projects link and experience in this field feel free to post some links. In the meantime I will read and Research a bit.
I think the idea would need a lot more hammock time though. If you were to try and build a whole app around it for example.
Normally, queues are used when concurrency is involved in some way. It makes sense if you think about what a queue gives you. It’s basically a place to buffer messages while you wait for something to be available which isn’t currently available.
I’d say, if that’s not the scenario you’re in, using queues would seem excessive and overly complex, like introducing accidental complexity.
There’s an intermediate between queues and function calls though. That’s the use of message passing over argument passing. Messages create looser coupling, because they only exchange data. Where as arguments can pass code or pointers around, basically things which require a context, which couples you to that context. This could be passing a function, or passing an object, or a reference to a shared variable.
A message forces you to serialize all the data needed by the receiver from the caller, at the time of the call.
Now, not all messages are like that. But messages can be like that. So you could have functions only take messages of that sort, probably won’t be the most efficient thing. On the other hand, that’s the idea behind micro-services. At the component level, you can have APIs that exchange such simple messages. Normally, these are expected to also run as independent process and communicate over a network. But I guess it wouldn’t need too. And those are more trivial to introduce a queue in between. I suspect this might be pretty close to how the Beam VM and Erlang works overall.
Off course, you’re trading something for that looser coupling, and that trade is not always for the best.
2 Likes
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.