[HELP WANTED] Release bundle size visualization


#1

I just added a new feature to shadow-cljs which generates a bundle-info.edn file which contains detailed information about :browser release builds.

The goal of this is to enable visualizations like this one:

Or any other kind of visualization you can come up with.

I have a demo bundle-info.edn (127 KB or huge gist). This is from a test build with a large number of JS deps and barely any CLJS so don’t expect it to resemble a real-world example. All the output is accurate though.

With [email protected] or later you can run shadow-cljs release your-build and find the file in target/shadow-cljs/builds/your-build/release/bundle-info.edn. Note that this is currently only enabled for :browser builds but could be done for all.

File Structure

The basic stucture of the bundle-info.edn file has two keys at the top

{:build-modules [...]
 :build-sources [...]

:build-modules

The :build-modules vector contains one entry per :module created for the build. A :module consists of the sources it included and the byte sizes. Each :sources entry is a “resource-id” refering to the :build-sources vector.

  {:module-id :base,
   :sources
   [[:shadow.build.classpath/resource "goog/base.js"]
    [:shadow.build.classpath/resource "goog/debug/error.js"]
    ...
    [:shadow.build.classpath/resource "cljs/core.cljs"]
    ...],
   :depends-on #{},
   :constants-size 1698,
   :js-size 142200,
   :gzip-size 29750}

:depends-on is a set of the :module-id it depends on. :constants-size is the raw size of all keywords/symbol constants in this :module as they are all collected from the individual sources to avoid repetition. :js-size is the raw :advanced byte size. :gzip-size is that JS gzip’d.

:build-sources

The :build-sources vector contains one entry per source file in the build.

 :build-sources
 [{:resource-id [:shadow.build.classpath/resource "goog/base.js"],
   :resource-name "goog/base.js",
   :module-id :base,
   :type :goog,
   :output-name "goog.base.js",
   :js-size 97323,
   :source-size 97323,
   :optimized-size 875}
  {:resource-id
   [:shadow.build.classpath/resource "goog/debug/error.js"],
   :resource-name "goog/debug/error.js",
   :module-id :base,
   :type :goog,
   :output-name "goog.debug.error.js",
   :js-size 1797,
   :source-size 1797,
   :optimized-size 0}
  ...
  {:resource-id [:shadow.build.classpath/resource "cljs/core.cljs"],
   :resource-name "cljs/core.cljs",
   :module-id :base,
   :type :cljs,
   :output-name "cljs.core.js",
   :js-size 1238250,
   :source-size 321953,
   :optimized-size 134156}
  ...]

:resource-id the one used in :build-modules :sources. :type may be

  • :goog for Closure JS
  • :cljs
  • :js for ES6/CommonJS
  • :shadow-js for node_modules

:optimized-size refers to the :advanced optimized filesize. A 0 may mean that all code was removed.
Note that this is not 100% accurate since the code may have been moved to other sources as well (Closure Code Motion). Its a pretty good indicator how much was removed/minified. cljs.core goes from 1.18MB -> 131KB.

The :shadow-js sources will not have an :optimized-size since it will not be optimized by :advanced.

The bundle-info.edn should contain all the information required to build a nice visualization. I may do this myself and some point but I have so much other stuff I want to do and could really use some help.

Ideally this would be done as a standalone library that just takes the bundle-info.edn as input and generates some kind of report as HTML/React.


Shadow-cljs - Build Reports
#2

To feed data to D3? https://bl.ocks.org/mbostock/4063423


#3

Yeah, d3 should work. No preferences really. Whatever works best.

https://gist.githubusercontent.com/thheller/ad9e09fb50b71f5ba7b7f96cfc58c1d1/raw/056c724b6dd96a533cb888d11c611ad2ab3b1ce3/stats.json

The obvious first solution was to just convert to the webpack format. You can drop that json into the webpage

http://chrisbateman.github.io/webpack-visualizer/

and get a pretty decent result.

source-map-explorer is another option.

https://code.thheller.com/data/source-map.html


#4

That’s a very cool idea and would be happy to help. The website links some D3 gist, which in turn brought me to a sunburst chart implementation for D3 v4.

Packaging it up for local consumption is probably what’s tricky here but I guess we could generate a static HTML page and load the JS “app” displaying it from a CDN. Or maybe that JS could be inlined into the HTML page removing any depdency/versioning issues. The user could then just open an .html file from their local disk and inspect the result.

I think it would be extra cool if you could compare previous build stats with your current one but that requires a bit more infrastructure for storing the results etc.

Can’t promise anything for next week but the one after next I should have some more time.


#5

Hmm source-map-explorer even maps the moved code back to the original location. So that would make it the most detailed option. I may just use that as it already does pretty much everything I’m looking for.

Not super pretty though.


#6

Packaging should not be a big deal since I want to display it in a UI webpage provided by shadow-cljs anyways. So I can just include any required .js (or even compile it on demand).

I like the idea of keeping a history so you can graph it over time. Should be ok to just write a file to a given directory with some kind of release tag (maybe even just a timestamp).


#7

Just pushed [email protected] which includes a new api/release-snapshot fn (not available via CLI yet).

It will create a snapshot in slightly modified format using the source maps to generate more accurate information which is also aware of code motion.

Each :build-modules entry will now contains a :source-bytes map

   :source-bytes
   {"cljs/core.cljs" 135294,
    "goog/array/array.js" 383,
    " [synthetic:1] " 75,
    "goog/math/integer.js" 4036,
    "goog/string/stringbuffer.js" 308,
    "demo/sm_test_before.cljs" 33,
    "shadow/cljs/constants/base.js" 147,
    "goog/object/object.js" 148,
    "goog/string/string.js" 163,
    "goog/base.js" 798}

The the bytes taken up by each source. Due to code motion a file may appear in multiple modules.

:optimized-size from the :build-sources is gone.


#8

Took the code from this gist and only made minimal adjustments, just wanted to see how it would look.

Its pretty but since you have to hover over everything not really useful.

Its a start though.

Also was too lazy to translate into CLJS so I just kept the JS.



#9

Since I can’t find a visualization I like I just added a simple table that expresses all the info I wanted to know.

With [email protected] you can do

npx shadow-cljs clj-repl
shadow-cljs - config: /Users/zilence/code/shadow-cljs/shadow-cljs.edn version: 2.0.101
shadow-cljs - connected to server
shadow-cljs - REPL - see (help), :repl/quit to exit
[11:0]~shadow.user=> (shadow/release-snapshot :ui {})
...

Then open http://localhost:9630 and follow the release snapshots links.

Looks something like this.

Far from perfect but enough to see how much each file contributes to the overall output.

Edit: (shadow/release-snapshot :ui {:tag "2.0.101"})). :tag defaults to latest which will override the previous one. Use :tag if you want something unique to make it comparable between releases.

Once I’m happy with this functionality I’ll add a proper CLI command.


#10

I’d vote for a tree-map viz, works great for showing relative chunks in a fixed width/height display. can also do nice drilldown if done using d3. BTW: try react-faux-dom (npm dep) if you are using d3 with react, it fixes all the problems of react vs d3 fighting over the dom.


#11

I just finished a revamped version of this which is available as of 2.3.6.

$ npx shadow-cljs clj-repl
(shadow/release-snapshot :app {})

Then open http://localhost:9630 and navigate the “Release Snapshot” links. I’ll probably add a button to skip the REPL step soon but for now you need to run it manually.

Looks like this

Everything is grouped by jar/npm package. Can expand to see individual files.

PS: Most of the work is done by react-table which was surprisingly nice to work with.