I’ve been drafting a proposal for using JavaScript’s parking primitives (e.g. yield, await) in ClojureScript, and it’s ready for more feedback!
There are some blockers right now, but would be interested in thoughts from community. Thanks!
5 Likes
There are still a few mistakes in the core.async example. I know what you are trying to say but you are misrepresenting core.async a bit.
(defn foo []
(let [ch (chan)]
(go
(doseq [i 100] ;; should be (range 100)
;; add a log here to see that it is only called 3 times
(<! (timeout 1000))
(>! ch i)))
ch))
(let [ch (foo)]
(take! ch) ;; should be <! in a go
(take! ch))
; foo is still running
The foo is still running is kind of incorrect. You are using an unbuffered channel which means the upper go loop will be parked until the >! until the can complete. The loop does not continue running while it is parked. If nothing ever takes from ch again and all references to it are GC’d the loop will be as well.
The behaviour is pretty much identical to generators, if you do not call .next() no further work will be done.
.return() on the other hand sets the generator into the completed state. Meaning that no further .next() calls can be done. This part can be done in core.async by closing the channel but the go loop would then need to check for it (when writing to it) but in return is then able to run code to clean up. You however do not need an extra control channel. I don’t think a generator itself can do anything if its terminated early.
(defn foo []
(let [ch (async/chan)]
(go (loop [i 100]
(when (pos? i)
(<! (async/timeout 1000))
(when (>! ch i)
(recur (dec i)))))
;; cleanup here
(prn ::terminated))
ch))
(let [ch (foo)]
(go (prn (<! ch))
(prn (<! ch))
(async/close! ch)))
Unless I’m mistaken the extra cleanup that can be done in core.async is not possible with .return. Not sure how much .return is actually used in practice since its pretty much the same as never calling .next again.
3 Likes
The document points out that IIFEs will cause a problem with yield and await:
(function(){ // <-- inserted by compiler
yield ...; // <-- error (must be inside function*)
await ...; // <-- error (must be inside async function)
}()
One suggested fix is to wait for the do syntax to land in javascript and use that.
But in the meantime, isn’t it simpler to just change the above to await (async function() { ... })() when you want to use await inside the function scope? I’m not as familiar with the yield feature, but that works for await.
Alternately, given that we are already emitting es6 code in this proposal, isn’t there a way of just using modern javascript’s lexical scoping instead of using an IIFE to emulate this feature?
Thanks, Thomas! Thrilled to see your simple core.async example and to learn that suspended go-blocks are garbage collected. Also, given that Clojure functions can’t have multiple exits anyway, I’m pretty satisfied with go-blocks having to handle the closed channel as a branching signal. 
Fixed in the proposal as just an interesting comparison now.
1 Like
Good proposal, though I’m unconvinced and agree with Nolen for now.
Nothing in your proposal showed me a use case or a blocker that justifies adding all this complexity. I worry if we put those in, that we will start seeing overly complicated async code in the wild.
From what I understand, there’s two issues.
- Interop
If you could show JS libs/frameworks that have Async generators in their API, and explain why its hard to work with, and show some example code. That would make a stronger case to me.
- Cljs async programming
I can see that a lot of people find core.async complicated and have a hard time using it properly. It might be that other async paradigms would be better. I think topics around that though should include Clojure. I wouldn’t want to see Clojure and ClojureScript shifting their async paradigms. I also think more people should just try promesa or use promises. I see a lot of people going to core.async too early. I see a lot of languages playing with different async paradigms, I feel we could wait a bit and learn from the experiments of others. In the meantime, I find promesa and plain promises fill the bill, and core.async is always there for advanced use cases.
I will preface my reply by noting that I am considering this from the perspective of javascript programmers coming to clojurescript rather than clojure programmers coming to clojurescript. With that in mind, take a look at the examples in puppeteer:
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://example.com');
// Get the "viewport" of the page, as reported by the page.
const dimensions = await page.evaluate(() => {
return {
width: document.documentElement.clientWidth,
height: document.documentElement.clientHeight,
deviceScaleFactor: window.devicePixelRatio
};
});
console.log('Dimensions:', dimensions);
await browser.close();
})();
Without some kind of async/await syntax, it is quite ugly to deal with this style of api. Also note that the evaluate function in these examples can also take a promise as an argument if what you are doing is async.
To use core.async, one must wrap each api call up in a channel rather than just calling the api directly. You also have to develop your own convention for how to handle errors, because promises have a separate error channel. If you want to pass a promise, you’ll write that in core.async and then wrap it in a promise. Note, this requires a user of the library not only to understand core.async but also to understand what exactly async and await mean, how they relate to promises, the differences in how the exception handling semantics work in both paradigms (not to to be underestimated), and how to translate all of that into core.async. On the other hand, if we had these features in clojurescript, translating from the examples is straightforward. Ergonomics matter.
One other point here: for non-clojure programmers coming to clojurescript, the documentation on core.async is challenging, since virtually all books, in-depth resources, and mailing list / forum discussions revolve around multithreading (!). I think a lot of experienced clojurians underestimate the difficultly in translating the core.async experience to clojurescript.
Honestly, I’d sooner use a library like the promesa library or from a library like kitchen-async. These macros tend to be a bit fragile, unfortunately, and they simply do not compose well. If you want to make 3 async calls in a row, they work. But if you want to do something complicated, they are awkward.
2 Likes
cool yeah, the do-expression syntax will work when available (only in stage 1 of 4 right now).
Converting the IIFE’s to async/generators might work. A lexical scope would be equivalent to an IIFE if a value could be returned from it. Maybe an outer temp var assigned inside the lexical scope could be used instead.
I disagree completely.
(go (let [browser (promise/await (.. puppeteer (launch)))
page (promise/await (.. browser (newPage)))]
(promise/await (.. page (goto "https://example.com")))
(let [dims (promise/await (.. page (evaluate some-js)))]
(js/console.log "dims" dims)
(promise/await (.. browser (close)))
)))
You can simply use :refer (await) to make this even nicer to look it but it absolutely no way looks “ugly”. Yes await has to be a macro to emit (<! (promise->chan (....))) but that is not a problem at all. You also don’t need any changes in the compiler to get this. Since you are controlling what the await macro emits you can also do just error handling the just throw failing promises or so.
core.async also scales to much more complex scenarios unlike promises.
Writing async code correctly is really really really hard and Promises only deal with a very small subset. It might appear like a good fit for some simple problems but feels pretty horrible otherwise.
Again: I’m not saying that core.async should be the goto solution for all async issues, it is not meant for that. There is a fantastic talk explaining that.
Another thing worth noting about Generators is that they do not have to be async at all. That is unlike core.async. In core.async after you >! or <! the next thing will be on the nextTick. Generators just let you write code in a sequential way but consume it on demand much like lazy sequences.
2 Likes
You’re missing my point. I get that it is possible for A+ programmers like you to recreate async/await syntax in clojurescript. My point is that there are libraries that really need that syntax in some form to be usable. If you try to reproduce the above example just using then/catch interop syntax, it is hideous.
Still, even if I were good enough to fill in the blanks of what you’re suggesting above, is it really a good solution to tell everybody who wants to use one of these libraries that they should first learn core.async (which they probably don’t need) and then go roll their own solution to interact with promises.
And even with the above solution, errors create a deep stack trace of generated code that is mostly useless for diagnostics. Even async stack traces in chrome have a maximum depth. Native async/await gives you stack traces in user code that are easy to understand. Native async/await will gives you all the corner cases of the language for free. Try catching two different kinds of exceptions in a go-block: it doesn’t work, and I don’t know how to fix it.
The naysayers’ argument sees to be that async/await isn’t strictly “necessary” to do anything in javascript. But core.async is complex library that is difficult to master, and, in my opinion, often tricky to debug in clojurescript. My argument is that native support gets you good tooling and guarantees that the language corner cases are exactly the same as the library you’re trying to interop with.
4 Likes
I’m not advocating against async/await support, quite the opposite. I’m just trying to argue that the argument should not involve core.async. I also don’t expect anyone to write the await macro themselves. Honestly I think it should be part of core.async for those already familiar with that. In the puppeteer example above you do not need to learn more about core.async than you’d need to learn about async/await though. You don’t have to do the super complicated stuff.
I do suspect that the core.async debug issue (probably just missing some source map data) and the catch issue are fixable. Someone just needs to dig into the code and do it. I will if I ever find the time but I don’t know how to do it either. I do enjoy a challenge though and the code interests me anyways.
1 Like
Sounds like async/await is the main ask.
Focusing on that would whittle the proposal to just:
(defn ^:js-async foo [] ;; async function
(let [x (js-await ...)] ;; await
(prn x)))
Is this easier to stomach if the less-important generator stuff is thrown out?
I suspect that the same amount of work is required in the compiler regardless, adding one additional thing that writes function* instead of function would be trivial.
I guess part of the debate is if we should build compiler hints, or special forms to allow the use of the underlying JavaScript coroutines and their syntax sugar, or if what we want is to add to Clojure/Script an async/await or other mechanism that would make working with Promises easier and arguably simpler.
I personally would rather we first explored the latter. What is missing in core.async to be satisfactory here? What’s missing from Promesa, what prevents another library to spawn and solve this problem?
core.async is an implementation of goroutines for Clojure mostly. Where each goroutine communicates with each other through channels. In Clojure, the goroutine runs over a thread pool, and are thus parallel and concurrent. As they park, the thread is freed to process another goroutine. Multiple goroutines can run in parallel up to your number of cores. Or you can spawn threads, and map goroutines one to one with them using thread. Basically, the whole machinery is promised on the idea that you will be parallelizing concurrent operations: non-blocking IO or short computations with go-blocks, or blocking IO and long computations with thread-blocks.
The examples I’m seeing here for JavaScript don’t have the goal to parallelize concurrent operations. They have the reverse goal, when using asynchronous APIs, which you don’t care to parallelize, how can you have a cleaner synchronous syntax?
I think this is the main reason why core.async doesn’t fit the bill for everyone. It is much more an alternative to Promises, whose goal is parallelization of tasks, where as async/await is a means to turn the async tasks synchronous again.
But, I don’t see what promesa lacks in that regard. Seems to me like it solves that problem pretty well.
Thoughts?
1 Like
Please consider my reply as one from a junior ClojureScript programmer. Most time of the day I write TypeScript and in our codebase there are lots of code using async/await. I think async functions is already part of the daily jobs among JavaScript/TypeScript programmers(at least in China).
I agree that core.async offers more powerful abstractions. However it’s really trivial now to write async function f(){ } and await f() to finish async tasks. Why do we want to translate the code into go blocks when a simpler way is already presented?
My example was using Koa. When I wanted to use Koa, I opened the docs of Koa, and translated the JavaScript into ClojureScript. It’s just too much work if I need to think about promises and go blocks, while with await it’s just adding two words:
// logger
app.use(async (ctx, next) => {
const start = Date.now();
await next();
const ms = Date.now() - start;
console.log(`${ctx.method} ${ctx.url} - ${ms}`);
});
1 Like
Can we write code like this today? Or is this part of this proposal?
Context: I am considering bringing CLJS into an existing Node.js application, initially as a set of libraries. I would like a nice way to avoid hugely nested callbacks and chained promises without too much extra baggage.
When you only want to just paper over the annoying single thread, everything async nature of Node, I’d like to do it without core.async, using just Promises.
It would be nice to have the option to bring in core.async when I have a use case that needs that kind of power. Perhaps though this appears almost immediately so the previous argument is moot?
Thanks @shaunlebron for taking the time to write all this down - it’s immensely helpful!
If the point is just to get async/await to handle JS promises, I think a few helpers make it very easy to handle, here is some snippets I use in some projects when I have to deal with JS promises, this is what the output code looks like:
(go
(let [json (-> (js/fetch "some-url.json") <!p (.json) <!p)]
(js/console.log "RESULT" json)))
And the implementation for it is quite simple:
(defn promise->chan [p]
(let [c (async/promise-chan)]
(.then p
#(async/put! c {:success %})
#(async/put! c {:error %}))
c))
(defn consumer-pair [resp]
(if (contains? resp :error)
(throw (:error resp))
(:success resp)))
(defmacro <!p [promise]
`(consumer-pair (async/<! (promise->chan ~promise))))
So IMO we don’t need anything extra on CLJS for that, maybe just create a library with some of those helpers. Propagating erros is something that could use some good shared defaults.
To make it fully “promise compat” we can have a function that wraps a go block and return a promise from it.
1 Like
I think that the syntax conversation can be solved in user-land for CLJS. Promesa covers that quite nicely, and other libraries could fill that in with similar macros if needed. That’s the beauty of lisps: you don’t need to change the core language to get new syntax. JS async/await works with promises anyway, so it is not like we are missing out on functionality by not emitting async or await.
I do think that having access to generators is important from an interop perspective, perhaps moreso than dnolen does, but I can’t argue with the fact that generators have not seen massive adoption in JS-land, making the value proposition for the work required fairly low…
I think it would be good to have a user-land implementation of a CLJS API for generators to see how it works and how it could be leveraged from CLJS. There are many interop use-cases that I think many people in the CLJS community are blind to / do not regard as important, that could be the difference between someone picking it up or not.
This discussion reminds me of why oget / oset never made it into core—I think the unspoken rules of ClojureScript are:
- Defer - never add what could be provided by a library.
- Partition - maintain strict disjoint sets of non-overlapping functionality.
- Reduce - completeness is not a goal—conservative over convenience.
For example, oget / oset was intended to make nested object access and assignment as easy as it is in plain JS, but it didn’t pass any of these rules.
This async/generator proposal passes some of #1, satisfies #2, but not #3, which I think explains a lot of the pushback here.
1 Like
I think it would help to have a real-world non-toy example that we can use to accurately compare all options and weigh pros/cons. We need a practical example where all the other options are clearly inferior to native async/await in JS, otherwise this probably goes nowhere.
Just adding things because JS did will probably get us into trouble at some point.
1 Like
I think the problem is that we disagree on what’s inferior. The discussion so far shows that our pros/cons lists don’t match each others, and that they’re also weighed differently.
The real world examples using actual libraries have been given by @jiyinyiyong and @jmlsf and their arguments are about convenience, familiarity and overhead—not about “just adding things because JS did”.
If I had to conclude this here, I’d say that if ClojureScript just favors conservatism over convenience (see #3 in aforementioned rules), then that’s the reason for rejection we should stamp on the proposal, which we can refer back to when this question comes up again in the future.
1 Like