Even though my own code only contains tens of modules, shadow-cljs has too load a bunch of files from Closure Library and Clojure standard library. Most of code can be dead code eliminated during optimization:
Although it’s cool when I develop my code with localhost, when I develop with IP address, it brings problems(not figuring out why, but there are too many requests):
Besides, sometimes I need to run my code in another devices, like phones, pads, the files goes over the network and then it’s loaded. Given the amount of files, it’s less convenient and in some cases it’s scary.
Yeah :loader-mode eval can solve major parts of this problem by merging all the files into a single file, then eval it. But there is still some issues that I have to wait and load a large file or use tricks to catch the exceptions.
So I do want to ask if there’s still any chance that we can remove the code we will never use, I guess I will not use goog.style.transition.js after it loaded. Then we can reduce the bundle size even in development.
DCE during development is simply not practical. The whole point of development mode is fast feedback which you simply cannot get if you try to analyze the code on every recompile to figure out what is “dead”. Also things that were “dead” at one point may not be later (eg. you add a new function that uses another function that was previously unused). Reconciling that at runtime would be a nightmare.
I also do not see the point in trying to micro optimize out small namespaces like the goog.style.transition you mention. It is used, otherwise it would not be loaded. Sure we can try and remove the code that uses it but overall it wouldn’t change much.
Don’t know what is happening with your webserver and why it fails to serve the files? Is this with the built-in shadow-cljs server or something else? I have never seen such issues.
The one thing that could be done to optimize download times is to allow caching, it can greatly improve overall performance if rarely changed files (eg. cljs.core) are cached. I would consider adding a flag that lets you disable the “no-cache” policy. I have personally been bitten by too many caching issues in the past so I wouldn’t recommend it but that doesn’t mean we can’t add it.
Another thing you can do is enable https which will make the built-in server switch to http/2 which can also greatly increase performance when many files are involved. You do however get better results overall with :loader-mode :eval + gzip.
The last thing you can always do is code-split and delay loading code you don’t actually use initially until you actually do. This is just as relevant during development as it is in production.
:loader-mode :eval + gzip has shown the best results in my tests and I have not come up with anything better so far. I’m all ears if you have ideas but DCE is not it.
I also do not see the point in trying to micro optimize out small namespaces like the goog.style.transition you mention. It is used, otherwise it would not be loaded.
I see so many files from goog.* so I think it contains all the files in Closure Library. Any means to tell if it’s really used?
Don’t know what is happening with your webserver and why it fails to serve the files? Is this with the built-in shadow-cljs server or something else? I have never seen such issues.
I was using the built-in server. The browser is Chrome canary. I debugging the page with http://<IP_ADDRESS>:7000. If I visit the page with http://localhost:7000, it’s all fine. I guess Chrome is banning some of the requests when there are too many.
The last thing you can always do is code-split and delay loading code you don’t actually use initially until you actually do. This is just as relevant during development as it is in production.
Not trying that since my own core base is relatively small compared to cljs.core and goog.*.
Another thing you can do is enable https which will make the built-in server switch to http/2 which can also greatly increase performance when many files are involved.
HTTP2 for files also means WSS for WebSockets. I have to think about that. It’s a solution but not very easy for me…
You do however get better results overall with :loader-mode :eval + gzip.
Agreed. It’s the best solution at current. 5M is large, but I can’t say it’s too slow over local network.
… use tricks to catch the exceptions.
When code is running in :eval mode, exceptions are caught, which means my breakpoints are not taking effects as usual. I can still turn on “Pause on caught exceptions”, just less easier to get the specific exception.
shadow-cljs only adds files to builds that were actually required somewhere. Whether or not something from those files is actually used is difficult to tell but they definitely were required by something. For example the goog.style.transition ends up included because of
(I’m trying to find a good visualization for this info so you can quickly tell why something was included in a build. A rough preview will probably be included in the next version).
I have never seen this issue even in test builds that had 3000+ included files. Do you have any plugins/extensions which may block those requests? Any configured proxy?
You mean exceptions thrown while loading the file?
I did some experiments and as of 2.7.22 enabled “soft” caching by default. This means that the browser is still forced to revalidate before using the cache but the server can reply with a “304 Not Modified” to allow skipping the actual download.
Depending on your network configuration and overall build composition this can outperform :loader-mode :eval. You may want to experiment a bit with this yourself since the results vary greatly. I still find :loader-mode :eval to have the best results in larger builds with lots of files (especially when accessing non-local servers).
Not sure if it’s useful but I recorded a video when I see the problem again. It’s fixed after I switched back to localhost:7000. https://youtu.be/ZEE6SHjL-7M
Is that your local IP or something running in a container or so? The server should basically never take that long to answer a file request. I’d suspect something inbetween you and the server. Might be a firewall, proxy, chrome extension, etc, etc …
Please try using a different webserver and see if it behaves the same. Any other webserver will do but should probably use a fast one (eg. nginx), otherwise it just might naturally throttle due to the webserver answering slowly.
