Storage backends for Datahike?

Hi, I’m playing a bit with Datahike, and I’m having issues using alternative storage backends. I tried both LevelDB and Postgres, and none of them seem to work.

I tried the configurations as described in the config docs but I only get errors. Using the file backend works as expected. Doing:

(:require '[datahike.api :as d])

(def config-postgres {:backend :pg :host "localhost"
                      :port 5432 :username "recipes"
                      :password "******" :path "/recipes"})
(def postgres-db (d/create-database config-postgres))

I get:

2. Unhandled clojure.lang.Compiler$CompilerException
   Error compiling NO_SOURCE_FILE at (8:18)
   #:clojure.error{:phase :execution,
                   :line 8,
                   :column 18,
                   :source "NO_SOURCE_FILE"}
             Compiler.java: 3707  clojure.lang.Compiler$InvokeExpr/eval
             Compiler.java:  457  clojure.lang.Compiler$DefExpr/eval
             Compiler.java: 7182  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   18  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   79  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   55  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  142  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  171  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  170  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  834  java.lang.Thread/run

1. Caused by java.lang.IllegalArgumentException
   Can't create a store with scheme: :pg

                store.cljc:   13  datahike.store$eval34960$fn__34962/invoke
              MultiFn.java:  229  clojure.lang.MultiFn/invoke
            connector.cljc:  147  datahike.connector$eval35179$fn__35186/invoke
            connector.cljc:   73  datahike.connector$eval35117$fn__35118$G__35104__35125/invoke
            connector.cljc:  183  datahike.connector$create_database/invokeStatic
            connector.cljc:  182  datahike.connector$create_database/doInvoke
               RestFn.java:  410  clojure.lang.RestFn/invoke
                  AFn.java:  154  clojure.lang.AFn/applyToHelper
               RestFn.java:  132  clojure.lang.RestFn/applyTo
             Compiler.java: 3702  clojure.lang.Compiler$InvokeExpr/eval
             Compiler.java:  457  clojure.lang.Compiler$DefExpr/eval
             Compiler.java: 7182  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   18  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   79  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   55  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  142  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  171  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  170  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  834  java.lang.Thread/run

If I try to use LevelDB, I get a similar error, saying it cant create a storage with scheme :level.

My deps.edn looks like so:

{:paths ["resources" "src"]
 :deps {org.clojure/clojure {:mvn/version "RELEASE"}
        io.replikativ/datahike {:mvn/version "0.2.1"}
        io.replikativ/datahike-leveldb {:mvn/version "0.1.0-SNAPSHOT"}
        io.replikativ/datahike-postgres {:mvn/version "0.1.0-SNAPSHOT"}}
 :aliases
 {:test {:extra-paths ["test"]
         :extra-deps {org.clojure/test.check {:mvn/version "RELEASE"}}}
  :runner
  {:extra-deps {com.cognitect/test-runner
                {:git/url "https://github.com/cognitect-labs/test-runner"
                 :sha "76568540e7f40268ad2b646110f237a60295fa3c"}}
   :main-opts ["-m" "cognitect.test-runner"
               "-d" "test"]}}}

I tried accessing the postgres DB with jdbc.next and it worked, so I think it’s not a setup issue, but I might be missing something else (the datahike docs are scarce). @konradkuehne, any ideas?

Thanks in advance!

Hi @mvarela, thanks for bringing the issue up. Could you check if it runs with [org.clojars.mihaelkonjevic/konserve-pg "0.1.2"]as another dependency? It seems like we have accidentally added "provided" in datahike-postgres dependencies. I’ll also check myself this configuration. Regarding the documentation, we are now building a separate product website for Datahike with all the docs and tutorials in one place. I’ll post it here once it is up.

Hi Konrad, thanks for the prompt reply. I had noticed the provided markers in datahike-postgres (BTW, you also have them in postgres-leveldb).

I’m still getting the same error after adding the konserve-postgres dependency. And I get a similar one for LevelDB. I’m fairly certain that right after you gave the Scicloj talk I had played with Datahike and LevelDB, but for some reason, it’s giving an this error now (I’m on Ubuntu BTW, I’ll give it a go on Mac and Manjaro later, just in case there’s some other underlying issue).

Hi, I’ve found the bug. An old hitchhiker-tree namespace was referenced and also an old datahike version was used. I fixed that and pushed a new datahike-postgres version 0.1.0-SNAPSHOT to clojars. Feel free to test it.

Besides datahike you need to require [datahike-postgres.core]as well since we need the functions from there for the postgres-interactions.

Ok, it now complains about some missing konserve class, but I suppose that’s related to the provided deps?. I’ll bring the ones listed in the datahike-postgres project.clj, and see if that helps. I’ll report back later.

deps.edn now contains this:

{:paths ["resources" "src"]
 :deps {org.clojure/clojure {:mvn/version "RELEASE"}
        io.replikativ/datahike {:mvn/version "0.2.1"}
        io.replikativ/konserve {:mvn/version "0.5.1"}
        org.clojars.mihaelkonjevic/konserve-pg {:mvn/version "0.1.2"}
        io.replikativ/konserve-leveldb {:mvn/version "0.1.2"}
        io.replikativ/datahike-leveldb {:mvn/version "0.1.0-SNAPSHOT"}
        io.replikativ/datahike-postgres {:mvn/version "0.1.0-SNAPSHOT"}}
 :aliases
 {:test {:extra-paths ["test"]
         :extra-deps {org.clojure/test.check {:mvn/version "RELEASE"}}}
  :runner
  {:extra-deps {com.cognitect/test-runner
                {:git/url "https://github.com/cognitect-labs/test-runner"
                 :sha "76568540e7f40268ad2b646110f237a60295fa3c"}}
   :main-opts ["-m" "cognitect.test-runner"
               "-d" "test"]}}}

but it’s still throwing:

2. Unhandled clojure.lang.Compiler$CompilerException
   Error compiling datahike_postgres/core.clj at (1:1)
   #:clojure.error{:phase :compile-syntax-check,
                   :line 1,
                   :column 1,
                   :source "datahike_postgres/core.clj"}
             Compiler.java: 7648  clojure.lang.Compiler/load
                   RT.java:  381  clojure.lang.RT/loadResourceScript
                   RT.java:  372  clojure.lang.RT/loadResourceScript
                   RT.java:  459  clojure.lang.RT/load
                   RT.java:  424  clojure.lang.RT/load
                  core.clj: 6126  clojure.core/load/fn
                  core.clj: 6125  clojure.core/load
                  core.clj: 6109  clojure.core/load
               RestFn.java:  408  clojure.lang.RestFn/invoke
                  core.clj: 5908  clojure.core/load-one
                  core.clj: 5903  clojure.core/load-one
                  core.clj: 5948  clojure.core/load-lib/fn
                  core.clj: 5947  clojure.core/load-lib
                  core.clj: 5928  clojure.core/load-lib
               RestFn.java:  142  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 5985  clojure.core/load-libs
                  core.clj: 5969  clojure.core/load-libs
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 6007  clojure.core/require
                  core.clj: 6007  clojure.core/require
               RestFn.java:  421  clojure.lang.RestFn/invoke
                      REPL:   49  fi.varela.recipes/eval35258/loading--auto--
                      REPL:   49  fi.varela.recipes/eval35258
                      REPL:   49  fi.varela.recipes/eval35258
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7166  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   18  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   79  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   55  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  142  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  171  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  170  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  834  java.lang.Thread/run

1. Caused by java.io.FileNotFoundException
   Could not locate hitchhiker/konserve__init.class, hitchhiker/konserve.clj or
   hitchhiker/konserve.cljc on classpath.

                   RT.java:  462  clojure.lang.RT/load
                   RT.java:  424  clojure.lang.RT/load
                  core.clj: 6126  clojure.core/load/fn
                  core.clj: 6125  clojure.core/load
                  core.clj: 6109  clojure.core/load
               RestFn.java:  408  clojure.lang.RestFn/invoke
                  core.clj: 5908  clojure.core/load-one
                  core.clj: 5903  clojure.core/load-one
                  core.clj: 5948  clojure.core/load-lib/fn
                  core.clj: 5947  clojure.core/load-lib
                  core.clj: 5928  clojure.core/load-lib
               RestFn.java:  142  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 5985  clojure.core/load-libs
                  core.clj: 5969  clojure.core/load-libs
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 6007  clojure.core/require
                  core.clj: 6007  clojure.core/require
               RestFn.java:  457  clojure.lang.RestFn/invoke
                  core.clj:    1  datahike-postgres.core/eval35264/loading--auto--
                  core.clj:    1  datahike-postgres.core/eval35264
                  core.clj:    1  datahike-postgres.core/eval35264
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7166  clojure.lang.Compiler/eval
             Compiler.java: 7636  clojure.lang.Compiler/load
                   RT.java:  381  clojure.lang.RT/loadResourceScript
                   RT.java:  372  clojure.lang.RT/loadResourceScript
                   RT.java:  459  clojure.lang.RT/load
                   RT.java:  424  clojure.lang.RT/load
                  core.clj: 6126  clojure.core/load/fn
                  core.clj: 6125  clojure.core/load
                  core.clj: 6109  clojure.core/load
               RestFn.java:  408  clojure.lang.RestFn/invoke
                  core.clj: 5908  clojure.core/load-one
                  core.clj: 5903  clojure.core/load-one
                  core.clj: 5948  clojure.core/load-lib/fn
                  core.clj: 5947  clojure.core/load-lib
                  core.clj: 5928  clojure.core/load-lib
               RestFn.java:  142  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 5985  clojure.core/load-libs
                  core.clj: 5969  clojure.core/load-libs
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 6007  clojure.core/require
                  core.clj: 6007  clojure.core/require
               RestFn.java:  421  clojure.lang.RestFn/invoke
                      REPL:   49  fi.varela.recipes/eval35258/loading--auto--
                      REPL:   49  fi.varela.recipes/eval35258
                      REPL:   49  fi.varela.recipes/eval35258
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7166  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   18  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   79  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   55  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  142  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  171  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  170  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  834  java.lang.Thread/run

Something seems quite weird…

I’ll have another look and get back to you. Maybe I’ll publish a clean datahike-postgres release version. Maybe the changes didn’t go through clojars.

Thanks! There’s no rush, though… the file backend works fine for what I’m doing now, I was just wondering what’s up with the others. I’ll be happy to do some more testing once you have a new release.

1 Like

Hi @mvarela, we fixed some further bugs and added the documentation to both backends with example code. Now we released a new version 0.1.0 for datahike-postgres you can check out.

Thanks, @konradkuehne, I’ll give it a go

Just tried it, and it works! Thanks a bunch!

1 Like