How to have a REPL-workflow for stateful interop?

Webdev_Tory · November 13, 2019, 2:50pm

My question is, what way have you found to you get to the benefits of the REPL fast-feedback cycle when dealing with stateful interop? To illustrate my question, my current protect: re-writing a Reagent view that renders a d3js chart. The final product will be a function with a long (let) block as it massages data (including converting to d3-compatible arrays), references and stores data about on-screen SVG object, and uses d3 to create a host of additional objects leveraging the objects all around them. DOM objects, color objects, scale bands, chart stacks, etc.

The difficulty here has two sides.

(let) statements are essentially black-boxes to the repl as they encapsulate state that can’t be referred to.
because this is Clojurescript with Figwheel and Reagent, breaking code due to a d3 mis-call or data-munge mishaps often necessitates a browser refresh which wipes out my dev state and has to reissue the AJAX calls that get me my original data.

So my development experience in very suboptimal. So far my attempt at improvement is to write it all using def statements and to rewrite it with let once it’s finished, but this is not going to be painless, either. Have you found ways to improve on an incremental workflow in high-state, high-interop situations?

mvarela · November 13, 2019, 3:11pm

I often do this when writing/debugging a complex function in Clojure itself. It works nicely, but you need to have some discipline for removing the defs and other cruft once you’re done.

A better solution might be @vvvvalvalval 's scope-capture, but I haven’t used it myself, so I can’t really comment.

Webdev_Tory · November 13, 2019, 4:30pm

Ah! Thank’s for sharing the scope-capture link. I remember hearing about something similar but couldn’t remember the name.

Anthony_Leonard · November 13, 2019, 4:34pm

I was going to reply pretty much the same thing as @mvarela . scope-capture is definitely worth a shot. IIRC using it does mean cluttering up live code when debugging though, which is why I quite like sticking to good old clojure.tools/trace when repl-ing a big fn that’s difficult to unwind (especially if I didn’t write it!). I wrote a little fork which makes the trace lib output in color using Puget, which wasn’t too hard to do (but still needed a fork for me )

witek · November 13, 2019, 7:54pm

Would be nice when the REPL or the IDE could do this spying automatically.

mvarela · November 13, 2019, 9:30pm

It could possibly be done in clj-refactor, or in the CIDER debugger, maybe with some namespacing so as not to clobber other vars in the ns. Not sure how much effort it would be, though.

lilactown · November 14, 2019, 2:13am

I was actually doing the exact same thing recently (reagent + d3) which is why I created a blog post about how I solved the “I am bad at d3 and blew up my web page” problem: Modern React in ClojureScript: Error Boundaries.

Using error boundaries let me fail fast and often without requiring me to reload the page ever.

When doing work that involves tweaking UI I prefer hot reloading + printlns over REPL, since I can see what the values I’m printing are and what the UI looks like at the same time. You can also split things out into helper functions when you have a particularly complex piece of logic that would allow you to develop it at the REPL and then integrate it with the larger component.

mvarela · November 14, 2019, 6:41am

That is very cool, indeed!

jiyinyiyong · November 14, 2019, 3:27pm

(let) statements are essentially black-boxes to the repl as they encapsulate state that can’t be referred to.

Might be out of topic but Chrome itself has a REPL in devtool and it provides access to objects printed in Console with js/console.log. Would that be helpful?

Webdev_Tory · November 14, 2019, 4:27pm

Good suggestion (though I rarely use Chrome). I don’t think that addresses the let problem, though, since they don’t always result in objects

Webdev_Tory · November 14, 2019, 5:32pm

This is purely BEAUTIFUL! Thanks!

lins05 · November 15, 2019, 8:59am

It’s definitely worth the time to add an error boundary wrapper in your top react element to avoid having to reload the page when the code is messed up (a typo, etc.).

One useful snippet is https://gist.github.com/pesterhazy/d163a8b3f1f1c6a0dac235858776c14b

greinseth · November 19, 2019, 3:14pm

Not really on topic, but might be useful:
You say you put a lot of dom-nodes created by d3 in your let-block? I would suggest not using d3 to generate dom nodes at all. Rather hand craft the svg with reagent elements instead and use d3 purely for the graph data. After we did this at my current project the view functions became much easier to work with and reason about.

Webdev_Tory · November 19, 2019, 6:55pm

@greinseth, I like the idea you mention, but I’m not sure how I would implement something like that. Here’s an example snippet of what I do to build our timeseries; you can see that there is a cascading amount of state that d3 creates/deals with. This is after I’ve started developing with def statements, but imagine it being in a let instead, as it should be in the end-product. How would your suggestion apply to stuff like this?

  (def series (.. g
                  (selectAll ".series")
                  (data d3-stack)
                  enter
                  (append "g")
                  (attr "fill" (fn [d i] (color (get years i))))))
  (def rectangles (.. (.selectAll series "rect")
                      (data identity)
                      enter
                      (append "rect")
                      (attr "x" (fn [d i]
                                  (x (nth years i 0))))
                      (attr "y" (svg-height svg margins))
                      (attr "width" (.bandwidth x))
                      (attr "height" 0)
                      (attr "id" (fn [d i]
                                   (let [id (str "rect_" i)]
                                     (str "hi")
                                     ;(attach-tooltip! id "Tooltip here?")
                                     id)))))

greinseth · November 20, 2019, 11:35am

Instead of using d3 to create the DOM elements, instead declare them in your reagent component. This way rendering and patching the DOM only happens by Reagent/React, your view components are greatly simplified (since they don’t have to branch out to the actual dom nodes).

I’ve created an example (in js, but the principle is the same) based on the Simple line graph with v4 example to illustrate what I mean:

My example code doesn’t draw axis ticks and labels. It is a little involved, but not as difficult as I first believed it to be.

greinseth · November 20, 2019, 8:57pm

I haven’t used d3 in ClojureScript before, but here is the same example in ClojureScript (more or less a direct port of the js code):

(defn line-graph [data]
  (let [arr (into-array data)
        margin {:top 20 :right 20 :bottom 20 :left 30}
        width (- 960 (:left margin) (:right margin))
        height (- 500 (:top margin) (:bottom margin))
        scale-x (-> (js/d3.scaleTime)
                    (.domain (js/d3.extent arr #(js/Date. (:timestamp %))))
                    (.range #js [0 width]))
        scale-y (-> (js/d3.scaleLinear)
                    (.domain #js [0 (js/d3.max arr #(get % :value))])
                    (.range #js [height 0]))
        line-fn (-> (js/d3.line)
                    (.x #(scale-x (js/Date. (:timestamp %))))
                    (.y #(scale-y (:value %))))]
    [:svg {:width (+ width (:left margin) (:right margin))
           :height (+ height (:top margin) (:bottom margin))}
     [:g {:transform (str "translate(" (:left margin) "," (+ height (:top margin)) ")")}
      [:line {:fill "none"
              :stroke "lightgrey"
              :stroke-width 2
              :x0 0
              :y0 0
              :x1 width
              :y1 0}]]
     [:g {:transform (str "translate(" (:left margin) "," (:top margin) ")")}
      [:line {:fill "none"
              :stroke "lightgrey"
              :stroke-width 2
              :x0 0
              :y0 0
              :x1 0
              :y1 height}]]
     [:g {:transform (str "translate(" (:left margin) "," (:top margin) ")")}
      [:path {:fill "none"
              :stroke "steelblue"
              :stroke-width 2
              :d (line-fn arr)}]]]))

(def example-data
  [{:timestamp "2012-05-01T00:00:00.0Z", :value 58.13},
   {:timestamp "2012-04-30T00:00:00.0Z", :value 53.98},
   {:timestamp "2012-04-27T00:00:00.0Z", :value 67.00},
   {:timestamp "2012-04-26T00:00:00.0Z", :value 89.70},
   {:timestamp "2012-04-25T00:00:00.0Z", :value 99.00},
   {:timestamp "2012-04-24T00:00:00.0Z", :value 130.28},
   {:timestamp "2012-04-23T00:00:00.0Z", :value 166.70},
   {:timestamp "2012-04-20T00:00:00.0Z", :value 234.98},
   {:timestamp "2012-04-19T00:00:00.0Z", :value 345.44},
   {:timestamp "2012-04-18T00:00:00.0Z", :value 443.34},
   {:timestamp "2012-04-17T00:00:00.0Z", :value 543.70},
   {:timestamp "2012-04-16T00:00:00.0Z", :value 580.13},
   {:timestamp "2012-04-13T00:00:00.0Z", :value 605.23},
   {:timestamp "2012-04-12T00:00:00.0Z", :value 622.77},
   {:timestamp "2012-04-11T00:00:00.0Z", :value 626.20},
   {:timestamp "2012-04-10T00:00:00.0Z", :value 628.44},
   {:timestamp "2012-04-09T00:00:00.0Z", :value 636.23},
   {:timestamp "2012-04-05T00:00:00.0Z", :value 633.68},
   {:timestamp "2012-04-04T00:00:00.0Z", :value 624.31},
   {:timestamp "2012-04-03T00:00:00.0Z", :value 629.32},
   {:timestamp "2012-04-02T00:00:00.0Z", :value 618.63},
   {:timestamp "2012-03-30T00:00:00.0Z", :value 599.55},
   {:timestamp "2012-03-29T00:00:00.0Z", :value 609.86},
   {:timestamp "2012-03-28T00:00:00.0Z", :value 617.62},
   {:timestamp "2012-03-27T00:00:00.0Z", :value 614.48},
   {:timestamp "2012-03-26T00:00:00.0Z", :value 606.98}])

system · May 21, 2020, 9:01am

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.