Async/Generator functions in CLJS - (requesting feedback!)

Questions & Help General Questions
9

You’re missing my point. I get that it is possible for A+ programmers like you to recreate async/await syntax in clojurescript. My point is that there are libraries that really need that syntax in some form to be usable. If you try to reproduce the above example just using then/catch interop syntax, it is hideous.

Still, even if I were good enough to fill in the blanks of what you’re suggesting above, is it really a good solution to tell everybody who wants to use one of these libraries that they should first learn core.async (which they probably don’t need) and then go roll their own solution to interact with promises.

And even with the above solution, errors create a deep stack trace of generated code that is mostly useless for diagnostics. Even async stack traces in chrome have a maximum depth. Native async/await gives you stack traces in user code that are easy to understand. Native async/await will gives you all the corner cases of the language for free. Try catching two different kinds of exceptions in a go-block: it doesn’t work, and I don’t know how to fix it.

The naysayers’ argument sees to be that async/await isn’t strictly “necessary” to do anything in javascript. But core.async is complex library that is difficult to master, and, in my opinion, often tricky to debug in clojurescript. My argument is that native support gets you good tooling and guarantees that the language corner cases are exactly the same as the library you’re trying to interop with.

4 Likes
10

I’m not advocating against async/await support, quite the opposite. I’m just trying to argue that the argument should not involve core.async. I also don’t expect anyone to write the await macro themselves. Honestly I think it should be part of core.async for those already familiar with that. In the puppeteer example above you do not need to learn more about core.async than you’d need to learn about async/await though. You don’t have to do the super complicated stuff.

I do suspect that the core.async debug issue (probably just missing some source map data) and the catch issue are fixable. Someone just needs to dig into the code and do it. I will if I ever find the time but I don’t know how to do it either. I do enjoy a challenge though and the code interests me anyways.

1 Like
11

Sounds like async/await is the main ask.

Focusing on that would whittle the proposal to just:

(defn ^:js-async foo []   ;; async function
  (let [x (js-await ...)] ;; await
    (prn x)))

Is this easier to stomach if the less-important generator stuff is thrown out?

12

I suspect that the same amount of work is required in the compiler regardless, adding one additional thing that writes function* instead of function would be trivial.

13

I guess part of the debate is if we should build compiler hints, or special forms to allow the use of the underlying JavaScript coroutines and their syntax sugar, or if what we want is to add to Clojure/Script an async/await or other mechanism that would make working with Promises easier and arguably simpler.

I personally would rather we first explored the latter. What is missing in core.async to be satisfactory here? What’s missing from Promesa, what prevents another library to spawn and solve this problem?

core.async is an implementation of goroutines for Clojure mostly. Where each goroutine communicates with each other through channels. In Clojure, the goroutine runs over a thread pool, and are thus parallel and concurrent. As they park, the thread is freed to process another goroutine. Multiple goroutines can run in parallel up to your number of cores. Or you can spawn threads, and map goroutines one to one with them using thread. Basically, the whole machinery is promised on the idea that you will be parallelizing concurrent operations: non-blocking IO or short computations with go-blocks, or blocking IO and long computations with thread-blocks.

The examples I’m seeing here for JavaScript don’t have the goal to parallelize concurrent operations. They have the reverse goal, when using asynchronous APIs, which you don’t care to parallelize, how can you have a cleaner synchronous syntax?

I think this is the main reason why core.async doesn’t fit the bill for everyone. It is much more an alternative to Promises, whose goal is parallelization of tasks, where as async/await is a means to turn the async tasks synchronous again.

But, I don’t see what promesa lacks in that regard. Seems to me like it solves that problem pretty well.

Thoughts?

1 Like
14

Please consider my reply as one from a junior ClojureScript programmer. Most time of the day I write TypeScript and in our codebase there are lots of code using async/await. I think async functions is already part of the daily jobs among JavaScript/TypeScript programmers(at least in China).

image

I agree that core.async offers more powerful abstractions. However it’s really trivial now to write async function f(){ } and await f() to finish async tasks. Why do we want to translate the code into go blocks when a simpler way is already presented?

My example was using Koa. When I wanted to use Koa, I opened the docs of Koa, and translated the JavaScript into ClojureScript. It’s just too much work if I need to think about promises and go blocks, while with await it’s just adding two words:

// logger

app.use(async (ctx, next) => {
  const start = Date.now();
  await next();
  const ms = Date.now() - start;
  console.log(`${ctx.method} ${ctx.url} - ${ms}`);
});
1 Like
15

Can we write code like this today? Or is this part of this proposal?

Context: I am considering bringing CLJS into an existing Node.js application, initially as a set of libraries. I would like a nice way to avoid hugely nested callbacks and chained promises without too much extra baggage.

When you only want to just paper over the annoying single thread, everything async nature of Node, I’d like to do it without core.async, using just Promises.

It would be nice to have the option to bring in core.async when I have a use case that needs that kind of power. Perhaps though this appears almost immediately so the previous argument is moot?

Thanks @shaunlebron for taking the time to write all this down - it’s immensely helpful!

16

If the point is just to get async/await to handle JS promises, I think a few helpers make it very easy to handle, here is some snippets I use in some projects when I have to deal with JS promises, this is what the output code looks like:

(go
  (let [json (-> (js/fetch "some-url.json") <!p (.json) <!p)]
    (js/console.log "RESULT" json)))

And the implementation for it is quite simple:

(defn promise->chan [p]
  (let [c (async/promise-chan)]
    (.then p
      #(async/put! c {:success %})
      #(async/put! c {:error %}))
    c))

(defn consumer-pair [resp]
  (if (contains? resp :error)
    (throw (:error resp))
    (:success resp)))

(defmacro <!p [promise]
  `(consumer-pair (async/<! (promise->chan ~promise))))

So IMO we don’t need anything extra on CLJS for that, maybe just create a library with some of those helpers. Propagating erros is something that could use some good shared defaults.

To make it fully “promise compat” we can have a function that wraps a go block and return a promise from it.

1 Like
17

I think that the syntax conversation can be solved in user-land for CLJS. Promesa covers that quite nicely, and other libraries could fill that in with similar macros if needed. That’s the beauty of lisps: you don’t need to change the core language to get new syntax. JS async/await works with promises anyway, so it is not like we are missing out on functionality by not emitting async or await.

I do think that having access to generators is important from an interop perspective, perhaps moreso than dnolen does, but I can’t argue with the fact that generators have not seen massive adoption in JS-land, making the value proposition for the work required fairly low…

I think it would be good to have a user-land implementation of a CLJS API for generators to see how it works and how it could be leveraged from CLJS. There are many interop use-cases that I think many people in the CLJS community are blind to / do not regard as important, that could be the difference between someone picking it up or not.

18

This discussion reminds me of why oget / oset never made it into core—I think the unspoken rules of ClojureScript are:

  1. Defer - never add what could be provided by a library.
  2. Partition - maintain strict disjoint sets of non-overlapping functionality.
  3. Reduce - completeness is not a goal—conservative over convenience.

For example, oget / oset was intended to make nested object access and assignment as easy as it is in plain JS, but it didn’t pass any of these rules.

This async/generator proposal passes some of #1, satisfies #2, but not #3, which I think explains a lot of the pushback here.

1 Like
19

I think it would help to have a real-world non-toy example that we can use to accurately compare all options and weigh pros/cons. We need a practical example where all the other options are clearly inferior to native async/await in JS, otherwise this probably goes nowhere.

Just adding things because JS did will probably get us into trouble at some point.

1 Like
20

I think the problem is that we disagree on what’s inferior. The discussion so far shows that our pros/cons lists don’t match each others, and that they’re also weighed differently.

The real world examples using actual libraries have been given by @jiyinyiyong and @jmlsf and their arguments are about convenience, familiarity and overhead—not about “just adding things because JS did”.

If I had to conclude this here, I’d say that if ClojureScript just favors conservatism over convenience (see #3 in aforementioned rules), then that’s the reason for rejection we should stamp on the proposal, which we can refer back to when this question comes up again in the future.

1 Like
21

Have everyone here actually looked at promesa’s async/await macro? I honestly can’t see why it doesn’t fully solve the problem?

Checkout this blog post for a simple example:

The only downside it you need to use bluebird promises, the benefit is bluebird promises are much faster then native JS.

1 Like
22

I must say, it is a bit frustrating that I keep pointing out the two main disadvantages of user-space solutions and then folks follow up with responses that ignore these points and reiterate the same points about user-space solutions. Feel free to disagree with my points, but ignoring them is super annoying. To respond that it is “possible” to code something that will work completely misses my two main points:

  1. Tooling is better with native solutions. Stack traces are shorter and more likely to point to user code.
  2. Corner cases, particularly with exceptions, are hard to get right. This is evidenced by the fact that neither core.async nor the promesa libraries get them right.

For example:

(p/alet [x (when true (await (p/delay 100 "foo")))]
      (println x))
;; =>
;; WARNING: Can't take value of macro cljs.core/when at line 7 /Users/justinlee/test/cljs-qs/src/tst/core.cljs
;; WARNING: Use of undeclared Var tst.core/await at line 7 /Users/justinlee/test/cljs-qs/src/tst/core.cljs

(try
  (throw (new js/TypeError "unexpected"))
  (catch ExceptionInfo e
    (println "outside async: ExceptionInfo catch"))
  (catch :default e
    (println "outside async: default catch")))
;; =>
;; "outside async: default catch"

(async
  (await (p/delay
           (try
             (throw (new js/TypeError "unexpected"))
             (catch ExceptionInfo e
               (println "inside async: ExceptionInfo catch"))
             (catch :default e
               (println "inside async: default catch"))))))

;; =>
;; silence of the lambs

I’m not complaining that these libraries have bugs. I couldn’t fix them myself, but I suspect people will be finding difficult corner cases until the end of time. I just think that if the native solution could be made to work, then we get to use the power of a billion eyeballs on the javascript implementations and save this smaller community’s resources to solve other problems. (For the record, I, personally, ran into these issues myself pretty much immediately upon trying to use them to port a serious piece of javascript to clojurescript.)

Also note @shaunlebron’s point about cancellable generators.

Finally, I get that there is justifiable angst about the javascript churn, but async/await is a syntax that came from F# in 2007, then went to C#, and only then came to Javascript. It’s essentially the same syntax as go and <!. I understand it’s in python, kotlin, and is coming to other languages like rust. This is a tried and true piece of syntax that is obviously a solid building block for event-based i/o.

5 Likes
23

I strongly favor conservatism in my language and convenience is very subjective. I’m still for adding async/await in general but the power of Lisp lets us do this in different places. Doing it at the compiler level requires much more thought than doing it in a library. I don’t know what the correct place is, just trying to discuss the options.

Async functions are contagious, once you go async everything has to go async. They are not the correct solution for every problem. Timothy Baldridge gave this excellent talk (already linked above) about the lessons learned with core.async. Those are the exact same lessons in async/await that the JS world has not learned yet. async function should be used just as sparingly as go. Doing async correctly is really really hard and just adding some syntax to make the easy case look easier does not help anyone. They are the “hot new thing” and are probably going to stick with us for a while though. Whatever happened to FRP (e.g. RxJS)? That used to be the answer to all of this?

Nobody is ignoring you. You are correct that the situation is not great. I do think that those are fixable issues however. At least in the core.async case, I have never used (and don’t like) promesa so I can’t speak for that. Yes, letting the JS world deal with the problems and not having to deal with them ourselves is certainly the easier path. That does not make it the correct path though. As with everything someone just has to go and fix it.

I think the Erlang crowd will disagree wildly with you here. The go (lang not macro) folks would probably disagree as well.

if all you have is a hammer, everything looks like a nail

Don’t dismiss other options just because JS has dictated that they are going that route. Yes, we should absolutely have an easy to use/understand interop option but that does not mean we have to use the exact same implementation.

1 Like
24

I’d love to learn what you are using for working in promises in a non-convoluted way. I have used nothing so far, and I’d like to hear some opinions :slight_smile:

25

Most of my own code uses core.async. Most the node interop is still callback based. I never had a case for await since my own code is otherwise already setup with channels in mind, so whenever I get a promise (e.g. js/fetch) the result is just put onto a channel which already exists. I find it much easier to think in terms of channels these days.

(-> (some-js/asyncFn 1 2 3)
    (.then one-thing)
    (.then another-thing))

is also completely acceptable to me.

26

I’m a bit late to the party, but looking at the debate from a distance makes it pretty clear that two problems are tackled at the same time (which is not the lisp way)

  • the asynchronous programming problem
  • the inversion of control problem

My main concern with javascript’s solution (hence with this proposal) is that it endorses promises at the syntax level, and promises (especially in JS flavor) are a very opinionated solution to a very hard problem. They promote imperative programming because they’re eager, they don’t support cancellation, and implicit flattening is just nonsensical. core.async has this problem as well because it forces ubiquitous usage of channels, which are opinionated as well and have their own drawbacks. Implementing error handling or cancellation on top of channels has overhead and feels clunky. go blocks are eager as well, making it inherently imperative. Clearly, asynchronous programming is not a solved problem and that’s why many developers have their own taste on which strategy is the best, making discussions emotive and subjective.

On the other hand, the inversion of control problem looks much less far from being solved and it would be really nice to see any effort to make it more generic and more solid in clojure. It makes sequential logic arguably more expressive than what monadic binding provides. The IOC mechanism used in the implementation of core.async go blocks is used in several other libraries, which is a good sign that the idea is good on its own. I have spent a fair amount of time myself digging into the internals to implement missionary’s sequential processes, and the amount of work required to decouple it from channel semantics is very reasonable. As a starting point, we could have it as a separate library and think harder about how to cover each situation (for instance, to have a less hacky way to throw an exception from an asynchronous parking), and fix some annoying corner cases such as nontrivial exception handling or self-hosted clojurescript support. Maybe in a second time we could think about improvements in the compilation stage to support host platform features such as javascript’s new syntax. This would be the lisp way.

5 Likes
27

Thanks. That syntax is also quite acceptable to me, but you do have to define one-thing and another-thing as functions. If the code is already written with this in mind, that’s fine — sometimes something like alet has its uses.

Do you use channels for Node callbacks as well? Essentially just using #(put! % c) as the callback?

28

I hadn’t really understood clearly your points, now I do. I’ll address them.

I think the major objection I have is at a higher level. You point out an implementation choice, like, how would ClojureScript implement async/await, and you say it could do so by leveraging ES8 (aka ES2017)'s implementation. I don’t disagree with that, if its possible to do so. If its a reliable implementation, you’re right, could make stack traces better and maybe its robust and already covers all edge cases.

But, I have an objection before we even go there, which is with adding async/await semantics to Clojure. Remember that ClojureScript is a Clojure to JavaScript compiler. Thus I feel any discussion to modify fundamental language semantics are a Clojure level discussion. So we should discuss that.

That’s because async/await is not a question of interop in my mind, since the actual things to interop with are Promises, and that’s already fully supported. Async/await are language level syntax. So really the debate is with adding async/await syntax to Clojure. And to be honest, when it comes to that, I think its Rich Hickey’s way or the highway. But I’m all for having an open conversation about it none the less.

So, if it was decided that Clojure would add async/await semantics, then maybe the ClojureScript implementation could leverage native ES2017 async/await. Though it would still need to transpile back to older ES targets, not sure if Google Closure can handle that.

In that perspective, I thus am not in favor of the proposal. This interop syntax will just end up being used like they are native Clojure syntax features I’m afraid. For now, without the use of libraries, Clojure doesn’t have any native mechanism for async modeling.

There’s an old proposal by Stuart Sierra to extend that: Home - Clojure Design - Clojure Development

It suggests adding callbacks to promises. This would then allow promises to be implemented in ClojureScript as well. Currently, it can’t, because they only work with multiple threads. If they were extended to support callbacks, you could use them in single threaded environments like ClojureScript. After a call to deliver a value to a promise, you just execute all registered callbacks in the order they were registered with the promise.

Now, back when the proposal was made, deref (aka @) was suggested to just throw an exception in ClojureScript if the promise is not yet delivered. But, given ES2017 async/await machinery, it might be possible to have deref implemented using await.

So, I’m not saying there’s no pros in implemented async/await with the native ES2017 solution. There could also be downsides to that as well, I haven’t really explored it. But I think the choice to add a async/await like syntax and semantic to the language must be discussed first.

1 Like