Switching out "dev" and "prod" environments with deps.edn

[zmitchell]

I’m learning web development at the moment and I’m using “Web Development with Clojure” as a resource, but the book is written around the Luminus framework. Luminus does a lot of things for you and is still based on Leiningen. I’m trying to learn things from the ground up and I’d like to learn how to build a project around deps.edn.

So, my actual question is around how you separate “dev” and “production” environments. Luminus has the <project>.env.<profile> namespaces that get switched out based on which profile you’re running the application under (dev or prod). How would you accomplish the same thing with deps.edn, and which kinds of things do you switch out based on development vs. production?

[dorab]

In the most basic case, you may need some extra (or different) dependencies in your project when you are in dev mode. For that, have an alias called :dev in your deps.edn that contains the extra dependencies. It is also common to add a dev directory to the :extra-paths key under the :dev alias. That way you can put resource files that are used in development in that folder. For example, you might want a different logging config in development than production.

For most other things, I generally prefer to have the same code but different configs (e.g., database type and location) for development versus production. I tend to use juxt/aero for my config management. It makes switching between different profiles very easy and yet just have one config file to deal with. If you use a config management system that uses different files for storing the config of different profiles, you can still stick the dev config file into the dev directory.

[zmitchell]

I guess my next question is how your code detects that you’re in “dev” mode. I’m imagining that there’s something in your codebase that does roughly this:

if dev_mode:
    load dev_config
else:
    load prod_config

Seems like a basic question, but I’m also a Clojure newbie

[gnarroway]

I usually use something like environ.

Each deployed environment (eg dev, uat, prod) is supplied with a key=value env file which works well both when running systemd units on a VM, or kubernetes deployments with configMaps.

For local dev, environ merges in Java opts that you can specify under an alias in deps.edn, so all the local connection strings etc are specified there.

An alternative I’ve seen is for your app to take a flag (eg env=prod) and pick up a prod specific config file (or values directly in namespaces) from your repo. In practice, separating operational concerns (eg runtime config) from your app source code is useful so would prefer not to do this. It’s also worthwhile reading about 12 factor apps

[dorab]

We were all clojure newbies once. And, tbh, it took me a while to figure out the same things you are struggling with. So I definitely empathize.

There is nothing explicit like what you suggest. The “magic” happens because there is a different CLASSPATH and deps.edn aliases in prod and dev. Also, since I am using juxt/aero (which I recommend) there is just one config that covers both dev and prod.

A while ago, I put up a pair of repos on github that has a barebones front-end and back-end that I used to learn some technologies. You might find these repos instructive to understand how the switch between dev and prod works in a deps.edn project. The repos are Hix (front-end) and Hax (back-end API server). For your needs, the Hax repo would be more appropriate.

Look at hax/resources/system.edn which is the config file. Note this excerpt:

{:port #profile
   {:prod 5555
    :dev 5554
    :test 5556}

What that says to juxt/aero is to use port 5555 for prod, 5554 for dev, and 5556 for the test environments.

The Usage section of the README shows you how to run the hax server in the dev and prod environments. Note that each invocation uses a different alias in the deps.edn.

Finally, if you look at hax.clj, you’ll see that, by default, the prod profile is used to set up the config.

If you look at user.clj, which is automatically loaded if it is on the CLASSPATH (which it is in the dev alias), you will see that the the dev profile is passed to the config processing in the set-prep! function.

Hope this helps. If not, please ask further questions.

[outrovurt]

I had the same problem a few years ago when I started learning web development in Clojure/Script. At the time I tried learning from a “Clojure Web Development Essentials” [Packt], which covers a much older version of Luminus. What I discovered was what you seem to be getting at: Luminus does not teach web development from the ground up. It’s more a template of defaults, and in fact I found that it actually hides a lot of important things from me. More than anything else, the book just introduced a tonne of dependencies and while it tried to explain each one, I didn’t come away with a good sense of how they all fit together.

I then chanced upon this tutorial:

http://clojure-doc.org/articles/tutorials/basic_web_development.html

(Sidenote: this site is brilliant for learning various aspects of the Clojure language.)

The above tutorial taught me everything about the core of what I need to know about: Ring. Once you understand Ring, you can understand how everything else which is built on top of Ring works.

Now, getting back to the meat of your question, as dorab mentioned, you basically need two things, a :dev alias, and the most important thing in the whole universe, a dev directory under your project root, containing at the very least a user.clj file which defines the user namespace. I can’t remember where I found out about this originally, I think it was in a post either by Stuart Sierra or referenced in one of his posts, but once I learned you could do this, everything fell into place afterwards. Let me elaborate a little.

My web projects are generally organized as follows:

• src/ - contains all server-side namespaces
• cljs-src/ - contains all CLJS code and .cljc files
• dev/ - contains user.clj which defines the user ns
• test/ - contains server-side and client-side tests, including UI tests

My deps.edn file will contain under :deps all the namespaces that the production website required. Under the :aliases section I will additionally define e.g. the following:

Excerpt from deps.edn

 :aliases
 {:fig {:extra-deps
        {
         ;; NOTE: Cider Jack-in CLJS includes piggieback implicitly;
         ;;  this is so that we can start it from a CLJ REPL;
         cider/piggieback                {:mvn/version "0.5.2"}
         com.bhauman/figwheel-main       {:mvn/version "0.2.14"}
         com.bhauman/rebel-readline-cljs {:mvn/version "0.1.4"}}}
  ;; This will feature extra data which we only want our rftest to pick up for now;
  :dev   {:extra-paths ["dev"
                        "test/core"
                        "test/unit"
                        "test/ui"
                        "test/main"]
          :extra-deps  {ring/ring-devel     {:mvn/version "1.9.4"}
                        etaoin/etaoin       {:mvn/version "0.4.6"}
                        lambdaisland/kaocha {:mvn/version "1.0.902"}
                        entmorph/test-utils {:local/root "../test-utils"}
                        }}
  :build {:extra-paths ["build"]
          :extra-deps  {io.github.seancorfield/build-clj
                        {:git/tag "v0.3.1" :git/sha "996ddfa"}}
          :ns-default  build}}

The :fig alias contains everything I require to start a figwheel CLJS REPL with hot-reloading. The :dev alias contains everything I require to launch my dev CLJ environment. I run these together, so from the command-line it would be:

clj -A:dev:fig

(I actually run this through emacs so I can have my REPLs accessible from within emacs. If you want me to elaborate on how to do this, please let me know.)

So, one important aspect of my :dev alias is the :extra-paths key which includes "dev". This is responsible for loading my user.clj file, which corresponds to the user namespace in which you start when you first fire up a REPL. So the big question is, what do you put in here? Really, anything you want which you only require for development. In my case, I define the following:

• functions to start, stop and restart a ring server
• a set of development/testing routes which are only for me to play around with and which should never make it to production
• functions for launching various figwheel builds, specifically one dev and once test version
• a function which I call from as part of a figwheel post-build hook which waits until the test figwheel build is finished and then runs some UI tests to make sure that none of the changes I have made in my client-code break anything

But the main thing to note is that your entire application entry point during development centres around this user.clj file and what you put in it. And the best part is that you can :require any namespace from your application and any vars from those namespaces, and pretty much spend most of your time developing from within the user namespace.

One more thing worth mentioning: you’ll notice I have included ring/ring-devel as a dependency under :extra-deps in the :dev alias. This is the correct way to use this library under deps.edn. Again, I only include it (and also for example reitit.ring.middleware.dev which comes from the excellent reititrouting library) in my user.clj.

And so this provide a clean separation between what is build in production, and what is built in development. In terms of environment, you can pretty much use EDN config files in the root of your project directory, or go for something like environ.

[fatmonky]

@RoelofWobben1 FYI too, wrt your other thread about free web dev tutorials for Clojure!

@outrovurt

I then chanced upon this tutorial:

http://clojure-doc.org/articles/tutorials/basic_web_development.html

(Sidenote: this site is brilliant for learning various aspects of the Clojure language.)

The above tutorial taught me everything about the core of what I need to know about: Ring. Once you understand Ring, you can understand how everything else which is built on top of Ring works.

[xbrln]

@dorab I am trying to figure out how to handle config and environment variables with juxt/aero in my leiningen project.

I created a config.edn file in resources folder and it looks like this

{:message "Hello"}

Then I required aero.core in the namespace where I want to access the config variable like [aero.core :refer (read-config)] and used it like

(def config (read-config (clojure.java.io/resource "config.edn")))

(:message config)

I was wondering if I have to do this in every namespace where I need the config, or is there a way to make config available in every namespace? So that I can avoid using (def config (read-config (clojure.java.io/resource "config.edn"))) in every namespace and just use (:message config) instead.

[dorab]

If you are using a lifecycle management library such as stuartsierra/component then use the config as a component, and pass in the config component as a dependency to other components that need it. That way, you don’t have a “global” and each component is passed in the config during creation via dependency injection.

If you are not using any such library, then, you could consider having a myproject.config namespace containing:

(ns myproject.config (:require [aero.core :refer read-config]))
(def config (read-config clojure.java.io/resource "config.edn")))
(defn message [] (get config :message))

and then in another namespace:

(ns myproject.other-ns (:require [myproject.config :as c]))
(defn myfunction (println "Message says" (c/message)))

Generally, I don’t like to have a def that has any significant computation during namespace load-time, but in this case, it might be okay.

[dorab]

Perhaps slightly better (in terms of testability and future-proofing) would be…

(ns myproject.config (:require [aero.core :refer read-config]))
(def config (read-config clojure.java.io/resource "config.edn")))
(defn message [conf] (get conf :message))

and

(ns myproject.other-ns (:require [myproject.config :as c]))
(defn myfunction (println "Message says" (c/message c/config)))

This would allow you to use a config that was provided in a different way (perhaps via stuartsierra/component) with fewer changes. It also follows the recommendations in aero recommendations