State management for the server?

Questions & Help General Questions
1

While working on refactoring notespace, I realized that it is gradually starting to face the typical state management problems of application development.

In this general, I am talking about this situation.

  • The app has a global state, held in one atom.
  • There are user events (e.g., the user calls some function at the REPL).
  • There are some other events (e.g., some future has been realized).
  • The effects of these events may be either updating the global state or some side effect (e.g., writing some file).

There are some nice libraries for such situations, such as Citrus and re-frame, but they are coupled with Rum and Reagent, respectively, and require clojuresript. There is also cljfx, which is coupled with JavaFX.

Has there been an attempt to write some general-purpose library for state management of this kind, that targets JVM Clojure?

(I might end up porting some part of Citrus, but that seems to require porting some parts of Rum as well.)

2 Likes
2

It seems like you realized this too, with your post edits, but it does look like citrus is less coupled to anything than it’s readme and project deps give it credit for, and should be able to be converted to cljc fairly easily

1 Like
3

Thanks @jjttjj!

1 Like
4

@thheller gave me a good answer for a similar question a while ago:

I think your case with Notespace is different, though. Notespace is doing simpler than “Generic backend app 3 with external connections and a database”.

1 Like
5

Thanks @teodorlu, interesting!

My question is not necessarily about global vs local state, but about using systematic event dispatch like thos of citrus, re-frame, cljfx, etc. – does anybody have good examples (and libraries) doing that generically on the server side.

1 Like
6

What exactly is the problem you are facing? I’m not sure I’m following?

2 Likes
7

Another related thing which is fully cljc is Domino: https://github.com/domino-clj/domino. Domino has been strange for me though. It seems good, it seems like it captures the essence of the important parts of state management. However I’ve always struggled to understand how to do stuff in domino the second I try to diverge from their basic examples (for instance, if I need to make an async request inside a domino handler which results in a callback which has data needed to change the state).

I actually have been looking for “general as possible” state management libraries myself and your post got me looking closely at citrus. And in messing with citrus I think I might be starting to “get” domino a little more. Perhaps domino attempts to be fully immutable where citrus attempts to provide the mutation layer? Maybe I need to (swap! ctx-atom domino/transact <changes>) from inside the async callbacks I have within my domino model.

I might have more thoughts on this later as I continue to think about this :slight_smile:

1 Like
8

I use re-frame often from CLJC; is there a particular part of it that doesn’t work for you without CLJS?

1 Like
9

@didibus Thanks, I am mainly seeking to hear about people’s practices (and favourite libraries) when working with async user events and mutable state on the server side.

10

Oh, forgot about Domino!
I will try to grok it too.
Thanks @jjttjj!

11

Thanks @Webdev_Tory! I had no idea that re-frame could be used as cljc. Indeed its code is mostly cljc. I thought it was coupled with reagent concepts such as ratoms and reactions. Rethinking about it, I guess it might be usable without them. Interesting!

Just ran across this 1-year old reddit discussion about re-frame on the jvm.

Do you use it for server-side state management?

12

I’m having a hard time imagining what sort of problem you’re trying to solve here. Mutable state on the server side seems like a bad idea to me, in general, unless you’re planning to have a single server process serving all clients, i.e., no scalability and no redundancy?

For me, “mutable state on the server” is a database :slight_smile:

2 Likes
13

Can you describe those async events? Is your server making async requests to other servers? Or are you trying to provide async APIs to clients of your server?

Maybe your use of server is confusing me? Do you have a server application, or just a desktop application?

A server receives requests and returns responses, generally over a socket, often to the web.

The asynchronous part would be that the client does not hold on to the connection until a response is returned, so the client would need to poll back for a response later or the client would need to be a listening to some callback port/socket and your server would call back the client when the response is ready.

In that respect, your server is still actively processing the request, or is waiting to process it.

14

Like @didibus and @seancorfield, I am unable to really understand your question without more details. Maybe you could narrow it down with some examples of the specific things you are doing and the ways in which it is proving difficult?

15

@seancorfield @didibus @jackrusher thanks.

About Notespace

What we are trying to create at Notespace is a thin layer of connection between the REPL+Editor UI to the browser. Its aim is to enable literate programming, testing, data exploration and visualization by means of gradually editing a static html document. The editing is done through REPL+editor interaction.

The generated document is a static html page such as these examples by @generateme: (1) (2).

The main question, currently, is what kind of REPL+edtior interaction would make editing such pages into a comfortable and dynamic process of fun exploration.

This problem is similar in its scope to org-babel, which is of course a source of inspiration. However, we seek a solution that would work in different editors, and without leaving the usual Clojure experience of editing a Clojure namespace. It is also related to Marginalia, but we seek to explore a more lively way of interaction, with a more complete support for data visualizations.

The current screencast demonstrates one possible answer to this question (see mainly the behaviour starting from 12:40). In this form, notespace has been useful to a few of us (mainly @generateme, who has been using it extensively at clojisr-examples).

But these days, we are considering some breaking changes. They will probably bring the Notespace experience closer to editing regular Clojure code, with much less special notation. So please expect some different concepts to arise very soon. Of course, comments and opinions will help. Discussions take place at the #notespace-dev Zulip stream.

About my question here

So, we are talking about a single-user desktop application, where the UI is the REPL and the editor, with the additional view in browser showing an html page. That page might be interactive, but currently not in a way that feeds back to the REPL state.

The app state that we have at our REPL process is mainly, for each namespace, the collection of notes (basicly, code blocks which are to be rendered together with their evaluation result). For each such note, there is some metadata (location at the file, type of note, etc.), and a state holding the last evaluation result. There is also some user configuration per namespace, and some additional information such as the namespace currently being edited, the last time each file was read, etc.

User events are function calls that take place at the REPL (e.g., “read all notes at this namespace”, “update the rendering of the note at this line”, etc.). Typically, they will happen through some editor key binding.

Things are happening concurrently: we have editor and REPL user interactions, and we have REPL evaluations, possibly running on different threads, that we want the view to react on their completion.

Since at this stage we wish to experiment with different interaction concepts, we need to write code that is refactored and extended easily,

Thanks

I hope that it answers some of the kind curiousity above.

For now, I will probably go with Domino mentioned by @jjttjj. If it turns out complicated, then maybe re-frame (on the JVM), or simple use of core.async pub and sub.

16

Thank you for the detailed explanation. It sounds like an interesting project and now I can see the context for what you are calling “server” in this – which is a single instance desktop application and now your question makes sense to me!

I look forward to examining the source code once you’ve integrated this new approach (but it’s mostly outside my area of experience so I don’t have much to suggest – I think this will just be a learning opportunity for me).

Have you considered having the browser panel integrated into the REPL+editor UI so it is one integrated app? I’ve gotten very used to working with REBL and viewing web pages within it, while working in the REPL (evaluating (java.net.URL. "http://domain.com") into a REBL-connected REPL will render that web page in the View pane, and then you can click and navigate right to drill into a link on that page: datafy and nav – this was something Stu showed off at Conj when he originally debuted REBL).

1 Like
17

Yes, I used word "server’ somewhat carelessly :blush:.

Good idea to look into integrating a browser panel!

18

One of the things which stuck with me reading Hoare’s CSP paper is that reading from a channel is like taking the value of a variable. Given this, you can implement a sort of local RPC process with channels, like so:

(defn reductions
  "Like core/reductions, but takes elements from in channel and
  produces them to out channel."
  ([rf init in out]
   (reductions rf init in out true))
  ([rf init in out close?]
   (a/go-loop
       [state init]
     (a/alt!
       in
       ([v]
        (if v
          (recur
           (rf state v))
          (when close?
            (a/close! out))))
       [[out state]]
       (recur state)))))

With this component you can always read the value of your state from out, and always update it with async events from in.
It then sounds like you’ll need a split between state-update-events and IO-events, have two CSPs reading from that split (one reductions and one simple sink loop).
This has the added bonus of eliminating global state and facilitating graceful shutdown.

1 Like
19

Some maria.cloud notebook features may be either directly helpful or may help by pointing to a similar class of solutions.

Primarily I’d like to share the dataflow abstraction, which we call cells, and which I believe are equivalent to your “notes”. Collections of notes, being a tree structure of related formula/value pairs, are precisely the killer use case for this abstraction. I wonder if a cells-like dataflow system might work better for your notebook needs than a system designed with client-server applications in mind.

The quickest visual explanation of cells dataflow may be this demo (timestamp 17:40) of Maria’s notebook dataflow. Matt Huebert’s When I Sit Down At My Editor, I Feel Relaxed is a longer and more precise explanation of the concept; for implementation see the cells directory of the maria repo.

Getting slightly further afield is Matt’s portion of the talk (timestamp 22:40) Of particular value:

  • :clojure: saving to “plain old Clojure files” (with a bit of Markdown in comments), no wrapping or special syntax
  • :books: all the special things we do (shapes, cells) are libraries that do not depend on Maria, so they are 100% portable to other applications

Hope this helps. :slight_smile:

1 Like
20

@bsless, thanks, that would be lovely to explore at some point! Maybe not on my first try, as I feel that I know a bit too little about CSP at the moment.

1 Like