Options for optimizing a re-frame+datascript application?

[tomc]

In my re-frame+datascript app, data loaded from the server goes into the datascript DB. I have re-frame subscriptions that query that database. Some of those queries can be slow for large data sets. The queries are typically not just d/pull or similar, but more involved, sometimes involving walking d/pull’s return value and doing additional work w/ the db for sub-entities.

The particular issue I’m seeing is that, because those queries depend on the datascript DB, whenever that DB is updated, the queries rerun, even if nothing changed that is relevant to the query at hand. For instance, my app has a UI where referenced entities can be opened in a “drawer” to edit the referenced entity. This loads the data for the referenced entity from the server into datascript, which causes all the subscriptions for the background UI to rerun, which causes sluggishness for the newly rendered UI.

I’ve tried breaking up the re-frame subs into multiple smaller subs, but that approach doesn’t help when the datascript DB is needed throughout the subscriptions.

I’ve tried optimizing the query logic, but I’m not sure how much more I can squeeze out. I’m using clojure.walk, which isn’t particularly fast in my experience, so perhaps I should use a custom function to walk the data?

Another option I’m considering is to maintain multiple datascript DBs, and only load data relevant for particular parts of the UI into each of them.

Have you dealt with this problem and come up with a good solution?

[p-himik]

Have you dealt with this problem and come up with a good solution?

Around 5 years ago, after testing DataScript’s performance, I ended up writing my own thing similar to GitHub - den1k/subgraph: Reactive graph database for re-frame. Nowadays, there are multiple alternatives, with some specifically designed to work with re-frame.

[tomc]

Thanks for the suggestion. Unfortunately, datascript is pretty deeply integrated into the app and that would be a major change.
I’ve found some performance improvements from using a filtered (datascript.core/filter) database for the subscriptions relevant to the page I’ve been having problems with. In order to get any more performance improvement I think I’ll either have to use your idea and change how the data is stored or simplify the UI.

[Harleqin]

You should try to layer your subscriptions. The subscriptions that are directly dependent on the app-db are always run on every app-db update, as you noticed. But you can make these as dumb as possible — just determine some relevant part of the app-db. Then, more involved subscriptions can use those as input. Re-frame automatically unifies and caches, i. e. if a subscription’s output doesn’t change, its subscribers are not even notified of the (non-)change.