Clojure Shells and Standard Output and CIDER

Questions & Help How to?
1

I’m trying to launch external programs from Clojure and have come across two libraries that work very well to various degrees:

Each provides a sh function for executing programs together with arguments and allowing options such as providing environment variables.

Where they differ is that clj-commons-exec allows you to specify an output stream to write to, whereas clojure.java.shell/sh just returns all the output in one go in the return map.

For example:

(sh ["path/to/program"] {:out System/out})

will redirect output to System/out.

This is particularly useful if I have a long running shell script and I want to see the output as it is printed in real-time, rather than having to wait until the whole script has finished running.

I can get clj-commons-exec to work from within a Clojure REPL launched via clj from a terminal, and I can also get it to work by launching a cider-nrepl server again from a terminal and doing e.g. lein repl :connect <port> using the port returned. However, when I try to do the same thing from within an emacs CIDER REPL, I only only see the final accumulated output rather than each message printed individually in real-time.

In case any of this isn’t clear, I’ve detailed it in an issue on the CIDER Github issues page, here:

It appears something is happening in CIDER but neither myself nor vemv are entirely sure what it might be.

I’m wondering if anyone has any insights as to what might be causing this, and if there is any simple way of fixing it or working around it that doesn’t involve rewriting/forking any existing libraries.

For reference, clj-commons-exec is a wrapper around Apache Commons Exec.

2

The clojure.java.shell is using nowadays dated java API:s for process control.

In the most recent dev-releases of clojure there is supported added for the newer Java Process API, in the namespace clojure.java.process.

https://clojure.org/releases/devchangelog

I suspect (but haven’t investigated) that Apache Commons Exec strives to address the problems with the dated java API used by clojure.java.shell, but that a similar functionality now is included in the Java standard library and used by clojure.java.process.

I hope the functionality in clojure.java.process is the solution you are looking for.

1 Like
3

Thanks for this. Bizarrely just yesterday I found out that this existed, after reading this excellent article by Sean Corfield on tools.build, and I got really excited thinking the same thing. But when I tried it, I got exactly the same problem.

Since I can get Apache Commons Exec to work in a normal terminal, my hunch is that there is something going on in CIDER, specifically in the way it processes standard output, but neither my Java nor emacs lisp are good enough to figure out what exactly.

4

I do apologize for my complete snafu, I was half-awake (and by extension half-asleep) when I read your post: I got clojure.java.process mixed up with clojure.tools.build.api/process, which also does something very similar to clojure.java.shell/sh and clj-commons-exec/sh.

So I really had to download the latest dev version of clojure to find it, and there is next to no documentation to go with it other than the docstring. I tried the following:

(start "./testing.sh")

and then:

(start {:out :inherit} "./testing.sh")

and have tried all kinds of values as the :out option, but no matter what I can’t seem to get it to work. I see the result is an ILookup (and an IDeref which returns the exit code), but even when I try to look at it using (:out result), I get:

#object[java.lang.ProcessBuilder$NullInputStream 0x750fdd88 "java.lang.ProcessBuilder$NullInputStream@750fdd88"]

Do you happen to know how I might use this function correctly, or if there are any resources available that might guide me?

5

There is a small test case:

This doesn’t answer your questions.

In the dev changelog: Clojure - Dev Changelog there is a link to the Jira ticket for this functionality where its design is discussed in some length: [CLJ-2759] - JIRA

The issues you are raising regarding the ILookup is discussed in the comments in the Jira ticket, but I’m not sure any solution is implemented yet.

But essentially the :out and :in are java ProcessBuilder.Redirects which are somewhat confusing, but there is some usage example in the ProcessBuilder Javadoc, which contains the following java code:

“Here is an example that starts a process with a modified working directory and environment, and redirects standard output and error to be appended to a log file:”

 ProcessBuilder pb =
   new ProcessBuilder("myCommand", "myArg1", "myArg2");
 Map<String, String> env = pb.environment();
 env.put("VAR1", "myValue");
 env.remove("OTHERVAR");
 env.put("VAR2", env.get("VAR1") + "suffix");
 pb.directory(new File("myDir"));
 File log = new File("log");
 pb.redirectErrorStream(true);
 pb.redirectOutput(Redirect.appendTo(log));
 Process p = pb.start();
 assert pb.redirectInput() == Redirect.PIPE;
 assert pb.redirectOutput().file() == log;
 assert p.getInputStream().read() == -1;

In the source code för clojure.java.process, clojure/src/clj/clojure/java/process.clj at master · clojure/clojure · GitHub , there is a comment with this code:

(comment
  ;; shell out and inherit the i/o
  (start {:out :inherit, :err :stdout} "ls" "-l")

  ;; write out and err to files, wait for process to exit, return exit code
  @(start {:out (to-file "out") :err (to-file "err")} "ls" "-l")

  ;; capture output to string
  (-> (start "ls" "-l") :out capture)

  ;; with exec
  (exec "ls" "-l")

  ;; read input from file
  (exec {:in (from-file "deps.edn")} "wc" "-l")
  )

Actually the object under :out is a java.lang.ProcessImpl$ProcessPipeInputStream, which derives from java.io.InputStream (as well as FilterInputStream and BufferedInputStream)

(require '[clojure.java.process :refer [start]])
(require '[clojure.java.io :as io])
(slurp (:out (start "ls")))

Slurp uses clojure.java.io to read everything from the InputStream and return it as a string, but you can use any methods of an InputStream to handle the out-data from the process in a streaming manner. The same goes for the :in key, but there it is an OutputStream, which can be wrapped in some suitable java.io.Writer, like java.io.BufferedWriter.

https://docs.oracle.com/javase/8/docs/api/java/lang/ProcessBuilder.html

1 Like
6

Thanks a lot for the extremely detailed feedback. At some stage I will sit down and try to get my head around it all. I think I’ll need quite a bit of time to figure out the way ProcessBuilder works and how this new functionality uses it, and by the sounds of things it’s still very new.

7

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.