Webworker as a library's dependency

[nek]

Hello people,

Is there any way to use a Web Worker defined in a library as a dependency? I guess not directly, but maybe there are some workarounds.

The question comes from the fact, that at the moment I have to use the library’s namespace as an entry point for the Web Worker in the main project, which in consequence requires including all the dependencies of the library in the main project.

[simongray]

Are you sure about that? Unused dependencies should be removed when making a release build if you use advanced optimisations (which you should).

[John_Newman]

I would really like this as well.

I went part of the way there with tau.alpha here

That functionality could be extracted out into a library specifically for programmatically spawning web-workers.

However, that was specifically designed to work with figwheel.main, with :none and :advanced compile modes. Haven’t tried it with shadow. I believe I’ve got it to work with vanilla cljs as well, IIRC, with different tweaks.

A comprehensive lib would work in all compile modes, in all cljs build tools, on web and node. And node is its own ball of wax.

I’m not sure, but coming up with such a lib may require agreement/coordination from cljs.main’s @dnolen, figwheel’s @bhauman and shadow’s @thheller, to ensure such a lib can depend on particular locations to draw goog, cljs, lib and app code from.

Ideally, this would work such that you could pass a root goog.module node to the spawn fn, like (web-worker/spawn {:name :monitor1 :goog-module :watch-dog-lib}), so each worker only grabs the goog.module dep graph it needs to satisfy its logic.

I’d be down to help spec it out and implement. Node is a bit of an unknown for me though.

With regard to libraries that expose capabilities that are backed by web-workers, even when using a spawn lib mentioned above, there are other concerns we may want to coordinate between libs to cleanly expose capabilities to all types of consumers in application code:

1. Will those capabilities be consumed by the application developer in the main thread or in yet another web-worker? In another web-worker, we have the option of exposing those capabilities as synchronous calls, whereas we don’t have that luxury in the main thread.
2. Are we working with persistent data-structures, js objects, or js primitives?
3. How will libs and apps want to handle memory - do we serialize via postMessage or via SharedArrayBuffers(SABs)? SABs require no communication overhead and no serialization overhead for primitive data. However, postMessage can be abstracted over clusters of machines.

For instance, if we wanted to build a leaflet-cljs library (similar to leafletjs), we could probably just use primitive data for the pixels, living on a SAB shared between workers for each tile being rendered. This would be much faster than via postMessage. However, if we wanted to do the rendering of much, much larger tiles, across a farm of servers, postMessage might be preferable.

Or, what if the application requires sharing js objects and cljs data structures between workers? Should a lib abstract away the serialization strategy? Should it also provide SAB and postMessage modes, for local vs distributed (multi-machine) processing?

[nek]

Thanks for the comprehensive overview! I felt like it was a pretty complex task but lacked the knowledge to reason about it. Your post gives me a broader context. In my case, I’ve gone with cljs immutable data structures and modded GitHub - jtkDvlp/cljs-workers: A clojurescript lib for performing async tasks via web workers. The data is serialized/deserialized using transit. This is the modded version of the lib cljs-audio/src/main/cljs_audio/workers at master · cljs-audio/cljs-audio · GitHub. Works well except for the problem we are discussing. Your idea of using goog.module sounds neat. Not sure if I’m down to driving the spec. Sounds like an interesting task with a huge benefit to the community, but I’m afraid I lack some critical knowledge both about ClojureScript and Web Workers. I’ll most probably revisit this problem, when/if my library becomes too cumbersome to use with manual Web Worker setup. Please ping me if you ever get to writing the spec.