One of the main on-line hangouts for Clojure people is the Clojurians Slack community. Unfortunately it suffers from its popularity. Slack will only retain the last 10,000 messages of history, which means that less than two weeks of logs are kept available.
A lot of valuable information is in that chat history, and so as a service to the community people set up a logging service back in 2015 so that these logs aren’t lost in time. (I believe it was Ruslan Prokopchuk , but if anyone can fill in the gaps in this history please do!)
In 2016 they were looking for someone to take over the hosting and running of the logs, and I agreed to do so, so since then it’s fallen on my shoulders to keep it running.
The setup so far was a bunch of python scripts, plus some node-based tools to generate the index pages, plus cron jobs and glue. I set up a VPS in 2016, got everything running, and left it do its thing. A few small improvements have been made since then, like making individual messages linkable, but mostly it’s stayed unchanged.
There were several things I always wanted to improve, for instance not all of the markdown that slack supports renders correctly, or it would be great to have links to navigate to the previous or next day of logs, but I’m not that good in Python and don’t really enjoy doing it, so these things never happened.
In November finally the site stopped updating. Because the amount of logs had grown so large (currently 1.7GB of JSON files and rising), some of the scripts were running out of memory. Messages were still being logged (luckily), but the site was no longer updating.
I decided that this was an opportunity to replace this with something that I myself could more easily work on and improve, and something that the community could more easily contribute to.
It’s all taken a little longer than planned, but at last I have some things to show. The app is on github as clojureverse/clojurians-log-app, it’s a simple Ring app, backed by Datomic. I was first going to go with plain PostgreSQL, but I figured using Datomic would make it more fun for me and for others to contribute. The flip side is that it’ll be a bit more costly to host, since Datomic performance relies on it having lots of memory to chew up.
The app isn’t perfect yet, but it’s working. A first version is up on the clojurians-log-staging.clojureverse.org domain. It’s quite slow right now, but there are still several things that can be done to improve that. Mostly though I’m planning to rely on CDN caching.
A big difference with the previous setup is that the complete setup and configuration is done through
ansible. You can check out my ansible-playbooks repo to see how things fit together. Deployment is done with a simple “git push”, similar to services like Heroku.
I think this can be a great project for people to contribute to, it’s small and self contained, but also has a real use. I’ve created a few issues of some of the most obvious things that still need doing on the app itself. I’d be happy to pair on these issues, or provide feedback and mentoring, especially if you still consider yourself a junior, or if you’re new to open source development.
There are still a few admin/opsy things that need doing before it can replace the main site, mainly setting up the CDN, and having it automatically import the new logs, but I should get to those soon.
Thank you all for your patience!