Hyperfiddle/rcf – turn your Rich Comment Forms into tests

Showcase
1

Releasing RCF: a repl-optimized test macro for Clojure/Script that turns your Rich Comment Blocks into tests (in same file as your fns). Send form or buffer to REPL to run tests and it squirts dopamine :white_check_mark: :white_check_mark: :white_check_mark:, very pleasing.

There are a lot of rich comment forms in the wild such as core.unify which are trivial to automate with RCF, just require the macro and use it. RCF is my first dependency in every new Clojure/Script project.

RCF is also a tool for thinking and communicating clearly with your colleagues. That’s why people write ;;=> in the first place – you can paste it into a chat in order to explain what the code should do. Boilerplate is the enemy of thinking and communicating.

Please try it! What do you think? Looking forward to your feedback!

image

https://github.com/hyperfiddle/rcf

5 Likes
2

I tried it and I like it!

It feels like a fast way to get an editor-agnostic REPL-based test runner, which is appealing for teams with different editor preferences. Seeing test results in a very visual way in the REPL is addictive. I like that tests live with the functions they’re testing (Rust does this for unit tests and it’s very easy to manage — no skipping between test and implementation files).

Some early comments after a quick try:

  1. Could you document how you intend users to run tests in example.clj in a REPL via the dev-entrypoint.cljc entry point so that the flag there is changed correctly?

    I’m using VS Code/Calva with deps.edn and nrepl in case that matters. I can run tests if they appear at the bottom of the dev-entrypoint.cljc file, but tests I run when reloading example.clj generate nil from the tests form. I’m missing something in my workflow to load the entrypoint file first. Sorry if this is really basic but I couldn’t figure it out after searching for info about REPL entry points or looking at the example/ dir.

  2. If you document it further, maybe it’s worth mentioning that dev-entrypoint needs to be a .cljc file just for those unfamiliar with reader conditionals. I think rcf will appeal to existing Clojure users but also to new users looking for a quick and easy way to write tests, so probably best not to assume too much knowledge up front.

  3. Are more descriptive test failures possible or worthwhile? Current output for a failed test:

clj꞉dev-entrypoint꞉> 
; Evaluating file: dev.cljc

❌ dev-entrypoint:15 

(hyperfiddle.rcf/unifies? (example/hello) "hi!")
 := 
false

nil

This tells me that the test failed but not why; it would be great to see more detail by capturing the actual output:

clj꞉dev-entrypoint꞉> 
; Evaluating file: dev.cljc

❌ dev-entrypoint:15 

(hyperfiddle.rcf/unifies? (example/hello) "hi!")
 := 
false

Expected: "hi!"
Got: "Hello, world!"

nil
1 Like
3

Hello Nick,

Thank you for the warm and detailed feedback!

About 1. and 2.

tests expands to nil by default. You certainly don’t want them to run when your REPL boots. In dev-entrypoint line 6 you can see how we enable it.
How we intend users to run tests:

  • start your repl
  • require rcf
  • enable it
  • eval tests forms at the REPL

The dev-entrypoint file isn’t mandatory or special, it’s just an example of how you might want to enable RCF in your existing app entrypoint. You don’t have to include it in your config, just copy what you might need from it.

About 3. You are right this error report is not helping at all. It’s already fixed but not merged on master yet. I’ll see if I can cherry-pick it.
Here is what the latest looks like:

(tests
 (str "hello") := (str "hi"))
;; =>

❌ example:33 
 in (str "hello")

"hello"
 :≠ 
(str "hi")

Also, RCF reporters are classic clojure.test reporters. You can customize/redefine them however you like. See reporters.clj.

4

I pushed some improvements to 3..

5

Thanks, Geoffrey! The failed test output is so much more helpful now:

❌ dev-entrypoint:15 
 in (example/hello)

"Hello, world!"
:≠ 
"hi!"

I appreciate the info about the clojure.test reporters too.

This is the part I’d love a little more guidance on. Everything works fine if tests are in the same file as the #'hyperfiddle.rcf/*enabled* lines.

Is that how I should be using RCF?

Or is there a workflow that enables RCF without putting (alter-var-root #'hyperfiddle.rcf/*enabled* (constantly true)) in the same file as my functions and tests (which feels like a bad idea, because they’d also run in production unless they’re removed)? So far I tried:

  1. Evaluating the dev-entrypoint.cljc file (with the *enabled* logic) and then evaluating tests in example.clj (this results in nil as output instead of test output for me).
  2. Evaluating the example.clj file in the REPL, running (alter-var-root #'hyperfiddle.rcf/*enabled* (constantly true)) directly in the REPL, (produces true as output) then re-evaluating the example.clj file in the REPL (which also produces nil instead of test output).
6

Yes definitely, you shouldn’t have to enable it more than once. I’ll provide more guidance and come back to you with a demo.

2 Likes
7

I got this to work with Calva like so:

A :repl alias in deps.edn with the :extra-path to a dev-only namespace; repl.cljc:

(ns repl
  (:require [hyperfiddle.rcf]))

; wait to enable tests until after app namespaces are loaded (intended for subsequent REPL interactions) 

#?(:clj  (alter-var-root #'hyperfiddle.rcf/*enabled* (constantly true))
   :cljs (set! hyperfiddle.rcf/*enabled* true))

; subsequent REPL interactions will run tests

; prevent test execution during cljs hot code reload
#?(:cljs (defn ^:dev/before-load stop [] (set! hyperfiddle.rcf/*enabled* false)))
#?(:cljs (defn ^:dev/after-load start [] (set! hyperfiddle.rcf/*enabled* true)))

(println "RCF enabled!")

Then in .vscode/settings.json:

  "calva.replConnectSequences": [
    {
      "name": "RCF",
      "projectType": "deps.edn",
      "menuSelections": {
        "cljAliases": ["repl"]
      },
      "afterCLJReplJackInCode": "(require 'repl)",
      "cljsType": "none"
    }
  ]

Select the RCF project type at jack-in.

The output window shows this when jacking in:

; Evaluating 'afterCLJReplJackInCode'
(require 'repl)
RCF enabled!
nil
clj꞉user꞉> 
; Jack-in done.
clj꞉user꞉>

After that loading files with (tests ...) forms run the tests. And you can also Evaluate Top Level Form in one of those (tests ...) forms to run them.

2 Likes
8

I’m lacking a way to check for inequality. Is that a candidate for inclusion?

9

By inequality, you mean < and > ?

10

I meant something like :<>.

11

My use case. I was using a 4Clojure problem (39) to see if I could leverage RCF for that. Seems I can. Though I would like to be able to check for the restrictions as well.

;; Define a function `f` which takes two sequences and
;; returns the first item from each, then the second 
;; item from each, then the third, etc.

;; Special Restrictions : interleave

(tests
 f :<> interleave
 (f [1 2 3] [:a :b :c]) := '(1 :a 2 :b 3 :c)
 (f [1 2] [3 4 5 6]) := '(1 3 2 4)
 (f [1 2 3 4] [5]) := [1 5]
 (f [30 20] [25 15]) := [30 25 20 15])
12

It’s an awesome use case! Let me check what I can do.

13

I pushed support for :<>, you can try latest master (09d546b56f6027fbea67736edf3af75acca29ba5)

1 Like
14

@nick, PEZ got it right, you either:

  1. enable it at the repl,
  2. put it in a dev-only file under an alias,
  3. use the -Dhyperfiddle.rcf.enabled=true JVM flag.

I’m using 2. for my dev setup. The same as PEZ, but for Emacs.
I’m using 3. to run tests in CI.

1 Like
15

Haha, that was the awesomest support I have ever experienced! :heart:

Trying to cheat the rules:

(def f interleave)

image
image732×530 28.3 KB

Thanks!

2 Likes
16

@PEZ @geoffreygaillard Thank you so much for your help and patience (and for the example, PEZ!). This is working perfectly for me now.

A silly gotcha for others: if you follow everything above and still see nil as output, double-check that your tests form is valid. I stupidly had (tests 1 :=1) instead of (tests 1 := 1) at one point, which gives nil instead of test output even if RCF is active.

I really like RCF and the workflow it enables! It feels very natural to formalise Rich Comments as a “real” testing system. Lots of people effectively write tests in Rich Comments as they work. Before RCF we had to leave Rich Comments at the bottom of our files like sad orphans, then go and rewrite largely the same experiments as formal tests elsewhere. I’m happy not to have to duplicate that work any more.

I wonder if there’s value in a form like this:

(tests-always
    "Always run these, even if `#'hyperfiddle.rcf/*enabled*` is false."
    1 := 1)

(Or tests-constantly or tests-enabled or similar.)

That way people could add RCF and write quick tests as experiments immediately without thinking about flags or reconfiguring their environment. They would have to accept responsibility for changing tests-always to tests before committing code or running it in production, but that might be a fair trade-off for a faster quick start.

Something like tests-always would also provide a way to selectively run a group of tests when evaluating a whole file if there are multiple tests forms in that file and I’m making changes to a function that also needs to be re-evaluated. (Just temporarily change tests to tests-always for one of the tests forms, leave the enabled flag as false, and re-evaluate the file.)

2 Likes
17

Thank you for your feedback everybody, it was harder than I realized to set this up, I have streamlined the configuration down to a one-liner for the readme to make it easier to get going.

3 Likes
18

This project inspired me to make an editor version of 4Clojure, utilising RFC:

2 Likes
19

This looks neat, and if you don’t mind, I’ll point out to a trend I’ve been seeing and I’d like to curb:

Project maturity: experimental, expect severe breaking changes.

I’m seeing more and more Clojure libs with this warning on their readme.

What happened to backwards compatibility matters, and no breaking changes!

A sign like that means even if it was awesome I wouldn’t use it for serious projects, but then, I’ve found, like on other libs before that were forever alpha, that they don’t end up changing and they work great…

So why all the fuss about it being experimental?

I’m hoping to see people release things in the future that they commit to not breaking users. Tell me, ya it works, we will fix major bugs, and if we need to change something that breaks existing users, we will release it as a new library or at least a new major version, or release the breaking changes under a new namespace, etc.

You don’t need to commit to long term support or anything like that, but if you’re not changing the API every month or faster, you’re way past “experimental” or “alpha”.

At least please eventually realize… Hey I guess I never changed the APIs and it’s been a year now… maybe I should update to a 1.0, a non-alpha, and remove experimental warnings.

\rent-over :stuck_out_tongue_closed_eyes:

3 Likes
20

Ok, sorry for my rent above haha,

Now I have a question/idea.

I like to sometimes as part of my doc-string show some example usage. So I wondered, and I’m not sure it’ll work, but could there somehow be a reader-macro something like #rcf/doc "" that would somehow return a doc-string so it works as one inside defn, but also runs the tests when it is evaled?

(defn double
  #rcf/doc
  "Multiply the given number by two.

   tests:
     positive:
     (double 2) := 4

     negatives:
     (double -2) := -4

     edge-cases:
     (double 0) := 0
     (double 1) := 1
     (double -1) := -1"
  [num]
  (* 2 num))

It would need to parse the string, but also I’m not sure how it could set itself up to run the tests, especially if like the function isn’t evaled yet at the time the reader macro expands… But if there was a way, I would find this awesome.

Edit:

I guess it could take a tuple instead, that might make it work with less trouble.

(defn double
  #rcf/doc
  ("Multiply the given number by two."

    (tests
      "positive"
      (double 2) := 4

      "negatives"
      (double -2) := -4

      "edge-cases"
      (double 0) := 0
      (double 1) := 1
      (double -1) := -1))
  [num]
  (* 2 num))

So #rcf/doc would transform the tuple into a string and return that, so the doc-string would include the tests in it so you see them when you look up the doc.

But now at the REPL I can also just eval the (tests ...) part which would run as a normal rcf tests macro.

And then it would need to somehow generate deftests at CI.

The only thing this wouldn’t do is run the tests in #rcf/doc when you load/eval the file or namespace. But I’d be okay with that compromise, unless there’s a way to get that working as well.

And I’m saying a reader-macro, because having a special defnrcf and the like is quite intrusive and doesn’t combine well with other libs that have their own defn like macro.

1 Like