Will the default be to return maps with namespaced keywords in the future? Is there support for other return types? I’m bit worried with the “qualify all the things” in the low level libraries, at least before Records can be optimized to support them. There was some discussion that the next version of Ring could also lean on namespaced keys(??).
About perf: to get close to the native java performance, we should do two things:
-
Precompile the queries: given an connection and optionally parameters, the query is just executed. All options, factories etc. are done in the (pre)compile stage.
-
Fast immutable query data: Persistent maps are much slower than Classes/Types/Records.
For an average low traffic enterprise app, fully dynamic response maps with qualified keys read from ResultSet
metadata are just great (and a good companion to clojure.spec
), but for high traffic (and high profile!) apps, we should be able use idiomatic clojure with ~Java-speed.
Did a small spike of porting some of the perf stuff from our libs into a minimalistic query-only jdbc wrapper: it allows statically populating return records, generating response records from (compiled) query, precompiling everything, including map responses etc. In the simple criterium micro-benchmarks, it seems to be 10x faster on query-side (including a roundtrup into the h2 memory database!). Not sure if I’m using next.jdbc
correctly yet, so the results might be totally bogus. Please correct if I’m using it wrong.
Anyway, I think that the clean separation of compilation and execution time would allow a single library to serve both use cases: fast tools for those (high-profile apps/startups) who need the perf and a “simple” api layer that just runs the compilation and the execution in a sequence, effectively giving the current api (and perf) of current jdbc.next
.
What do you think? How to proceed? Big internal refactor PR to jdbc.next
? Something else? Would be great to have just one solid and maintained solution for the community, and outside of the contribs.
Quick results, a modified criterium suite:
(defrecord Fruit [id name appearance cost grade])
;; java-fast mapping
(defn rs->fruit [^ResultSet rs]
(->Fruit
(.getObject rs 1)
(.getObject rs 2)
(.getObject rs 3)
(.getObject rs 4)
(.getObject rs 5)))
;; 680ns (vanilla java)
(bench!
"java"
(java-query "SELECT * FROM fruit" con))
;; 690ns (hand-crafted function to populate the record)
(let [query (p/compile "SELECT * FROM fruit" {:row rs->fruit})]
(bench!
"p: manual, record"
(query con)))
;; 710ns (infer code of rs->fruit from a given record class)
(let [query (p/compile "SELECT * FROM fruit" {:row (p/rs->record Fruit)})]
(bench!
"p: derived, record"
(query con)))
;; 710ns (create a new record for the unique resultset using compile-time inspection
;; + a positional constructor for it, memoized + macro / eval => know what you are doing!)
(let [query (p/compile "SELECT * FROM fruit" {:con con :row-gen p/record-row-gen})]
(bench!
"p: compiled map"
(query con)))
;; 1800ns (precompiled fully dynamic map result, just like next.jdbc / java.jdbc)
(let [query (p/compile "SELECT * FROM fruit" {:con con})]
(bench!
"p: dynamic map"
(query con)))
;; 3000ns (fully dynamic map result, just like next.jdbc / java.jdbc)
(let [query (p/compile "SELECT * FROM fruit")]
(bench!
"p: dynamic map"
(query con)))
;; 6700ns
(bench!
"next.jdbc: reducible!"
(into [] (map (partial into {})) (j/reducible! con ["select * from fruit"])))
;; 7000ns
(bench!
"next.jdbc: execute!"
(j/execute!
con
["select * from fruit"]
(partial into {})
{}))
;; 8000ns
(bench!
"java.jdbc: query"
(j1/query {:connection con} ["SELECT * FROM fruit"]))