Datomic - datomocking/datoforking

[alexandrkozyrev]

Here’s a quote from

GitHub - vvvvalvalval/datomock: Mocking and forking Datomic Peer connections...

Rationale and semantics

Mocked connections use Datomic’s speculative writes (db.with()) and Clojure’s managed references to emulate a Datomic connection locally.
The main benefit is the ability to ‘fork’ Datomic connections. More precisely, if conn1 is forked from conn2:

• at the time of forking, conn1 and conn2 hold the same database value;
• subsequent writes to conn1 will leave conn2 unaffected
• subsequent writes to conn2 will leave conn1 unaffected

Let’s say you work on a problem and, as usual, you have several implementation ideas (aka Plan A, B, C…) and unpredictable implementational “I believe I did, Bob” moments.

So you start with Plan A (and try at the same time to cover common parts with other Plans, if any) and push until you stuck at some point. You switch to Plan B, just to find out that the key technology has unacceptable pricing model. So you switch to Plan C. Somewhere in the middle of Plan C you somehow find a solution for Plan A problem point and switch back to Plan A. And finish you task with Plan A.

During these back-and-forth motions, you’ve found a very useful tool (during Plan C), and have changed key element of Plan A, based on Plan B experience.

Back to Datomic.
What is(are) a good way to store these pivotal points (datomic forks/mocks) in Datomic?
It looks like “All values are equal, but some values are more equal than others”.
The very first version of this question: how to store datomic.db.Db@b1dbbe41 in Datomic?

[vvvvalvalval]

Author of Datomock here.

First, very concretely:

@alexandrkozyrev

how to store datomic.db.Db@b1dbbe41 in Datomic?

Since a Datomic Database value is not a legal Datomic value type, you can’t store one in a Datom.

What’s more, in spite of the popular Git/Datomic analogy, you can’t make several parallel versions of a Datomic database coexist in a Datomic system the same way you can have several parallel versions of a codebase coexist in a Git repository (and be reconciled later).

What you can do is keep several backups of a durable Datomic database representing “alternative versions” of how the data evolves, and switch between them using restores (this tends to get impractical as the size of the data grow, because backups / restores will take long).

Now, regarding your general approach (I may have misunderstood what you’re after, but I’ll give it a try):

I’m not sure what you’re after is sensible. The way I see, the role of a Datomic database is primarily to store information, not decision-making logic (the decision-making logic is in your application code). Even stored transaction functions are not here to express what decisions should be done, but what decisions can be made.

Therefore, IMO, your various alternatives (Plan A / Plan B / Plan C / the original point they come from) should not reside in a Datomic Database, but in your Git history.

Now, of course, evolving to Plan A / Plan B / Plan C may require to add stuff to the database, for instance data migrations, new attributes, new transaction functions, etc. This is where Datomock (or the lower-level speculative features of Datomic) are helpful: because they enable you to experiment with these evolutions without committing to them, via cheap ephemeral local in-memory forks.

[alexandrkozyrev]

Valentin, I will use universe term (aka context, environment), like in you post https://vvvvalvalval.github.io/posts/2016-01-03-architecture-datomic-branching-reality.html

Hal Abelson said that he can not answer “sameness” question in less than an hour.

I think it’s a “sameness definition” question.

Let’s say I want to have several universes in Datomic.
Here are my options for several parallel versions of “the same” universe:

1. several backups of Datomic database
2. several partitions in Datomic database
3. additional attribute for each entity
4. several Datomic Database values (somewhere, somehow)
5. several Datomic databases (new database for new universe)
6. several mega-entities (registers) to log parallel versions evolution

Any recommendations/comments on these options?
If this https://groups.google.com/forum/#!topic/clojure/NB3d5w-7M3o is a good option for Clojure’s data structures implementation, what is a good option for datoforking in Datomic?

[vvvvalvalval]

Sorry, I fail to make sense of what you’re asking; we may have a different notion of what Universe means (in my post, Universe is a mutable abstraction representing the environment to which domain logic is applied).

I get the feeling that you are trying to treat information in Datomic a bit like code in Git, with parallel branches coexisting durably, in which case my answer would be: Datomic is not a great fit for that, and I don’t think it’s a desirable thing, due to the essential differences between code and information.

Maybe if you could express it in a more formal way…

[dustingetz]

I think Datomic could be like git, because values should be referentially transparent, fully independent, no spooky action at a distance. See the gist of spooky action at a distance in datomic “values”, linked here: https://forum.datomic.com/t/interactions-of-d-basis-t-d-as-of-d-with/219 For all Cognitect’s rhetoric about database-as-a-value, it really caught me by surprise that Datomic databases are “value-like in many ways” but not ACTUALLY values. I dont think Rich Hickey would complect something like this without a really good implementation reason, and I’d like to understand better what that is. Or maybe I have got something wrong.

So i’m not sure if what Alex is trying to do is possible today.

[dustingetz]

One thing that I think is possible is to basically always rebase your branches - you can maintain your own “branches” with d/with, so long as the branches are always rebased onto the latest known t, with no as-of or other database filters. So no branching from the past. I might have this wrong though.