Library documentation: the Cljdoc / Notebook tension

Cljdoc is amazing for API docs. One great example is rewrite-clj’s Node API-doc. For guides, it can show Markdown.

For guides, I prefer to author notebooks. And as a reader, I find that full notebooks as documentation can help shift the focus from tool to problem at hand. Both Clerk and Clay/Quarto can yield great results, see for example nextjournal/markdown (Clerk) and Noj (Clay/Quarto).

To summarize,

  • A clean index of public symbols (ala cljdoc) is great for API Docs
  • Notebooks are great for explaining problems.

Details and problems.

We, the Clojure community, commonly publish library documentation with a build step. We check in Clojure and Markdown, and have the build process render our library docs. Cljdoc currently does that automatically for libraries, with some help of an EDN file. That process can fail, and can’t take too long. Clojure Civitas currently rebuilds every Clojure namespace on every push, meaning slow notebooks will slow down deploys for everyone.

Given the publishing differences and limitations outlined above, I fail to see “the answer”.

Proposed goal.

But problems are soluble. To me, the technical differences and difficulties don’t seem impossible to overcome. In my opinion, we’d like:

  1. Notebooks to explain problems and solutions
  2. Clean indexes of public symbols for API documentation
  3. Good, near-instant local preview of library documentation
  4. As light as possible build step for publishing library documentation

I think it’s possible, but I fail to see the best path to get there.

Ideas?

2 Likes

Another data point is dedicated, custom documentation websites. Replicant’s documentation is one such example: https://replicant.fun/

It gains some nice UI fragments, such as the ability to copy code from tutorials. And it can be thematically consistent: logo colors play nicely with website colors.

Or maybe browser UI libraries just play in their own special category.

For many of our libraries, we use Codox, and like it.

Here’s an example: TMD 8.001

In addition to api documentation, codox has the concept of “Topcs”, which I suspect might be an interesting point of integration with Clay.

I think the questions you raise are good ones, and look forward to seeing others thoughts.

I think this post has pretty good advice about documentation. Ideally, you have tutorials, guides, topical explanations, and references.TMD’s docs are a great example. I think it’s also helpful to have examples and great search.

I use both clerk and codox for documentation (I also copy TMD’s theming :stuck_out_tongue: ) . See llama.clj for an example.

I’ve been working on collecting searchable docs with dewey. You can see some example searches here and here. Both of these queries are powered by a sqlite database that gets exported regularly.

From an indexing perspective, I think one area we could improve as a community is having a standard way to point to docs, examples, references, etc. For example, if there was a docs.edn with some keys that linked to relevant info. Specifically for examples, it would be great if there was a datafied way to lookup typical requires, necessary :deps coordinates, and some sample usages similar to clojuredocs.

I’m not sure what you mean by “light”, but for me, it’s totally fine for notebooks to take a long time to generate. The notebooks for llama.clj take 13 minutes to generate, but that’s not really an issue since I can run it asynchronously in a github action. If something isn’t working, I can execute the notebook locally step by step. What do you mean by “light”? Why is it important?

I don’t think it’s a problem that notebooks are slow. I think the problem is that notebooks are rebuilt even when no changes have been made (I’m not trying to fault Civitas’ approach since making a deployment system takes time and energy to make). Rather than focusing on making notebooks all notebooks faster, I think making the deployment more scalable can help growth as well.