Why doesn't my program exit?

[geokon-gh]

I’m trying to use Clojure to write a quick little script but my program won’t exit for some reason.

(ns benchmark.core
  (:require [clojure.java.shell :refer :all])
  (:gen-class))

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (let [command "ls"
        args "-al"
        program-run (sh command args)]
   (println  (:out program-run))))

The weird thing is that In the REPL (Emacs/CIDER) I can run (-main) and it’ll print the output and then go back to waiting for the next user provided command to run. But when I run it with lein run it’ll just hang after printing.

However, I’ve confirmed that a simple (println "Hello World") exits properly

I’m not even sure how to go about debugging this. Any input would be helpful

[teodorlu]

This sounds like an issue I encountered a while ago:

Issue: `lein repl` times out with no useful error message when using `clojure.java.shell/sh` in user.clj

[geokon-gh]

Thanks for that @teodorlu . Glad I’m not alone
Clojure keeps throwing me suprises

I’ve found a work around for the time being. Adding a (shutdown-agents) at the end seems to force termination

(ns benchmark.core
  (:require [clojure.java.shell :refer :all])
  (:gen-class))

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (let [command "ls"
        args "-al"
        program-run (sh command args)]
    (println  (:out program-run))
    (shutdown-agents))) 

Let me know if you have some kind of better solution. I’d like to add a note to Clojuredocs so that no one else needs to get stumped by this

[pesterhazy]

clojure.java.shell/sh uses clojure.core/future

Compare:

$ clojure -e '(println "hi")'
hi
$ clojure -e '(future 1) (println "hi")'
#object[clojure.core$future_call$reify__8097 0x341814d3 {:status :ready, :val #object[clojure.core$identity 0x3f67593e "clojure.core$identity@3f67593e"]}]
hi
^C

# had to interrupt using ^C

$ clojure -e '(future 1) (println "hi") (shutdown-agents)'
#object[clojure.core$future_call$reify__8097 0x341814d3 {:status :ready, :val #object[clojure.core$identity 0x3f67593e "clojure.core$identity@3f67593e"]}]
hi
# returned immediately

[pesterhazy]

Also see https://dev.clojure.org/jira/browse/CLJ-124

[teodorlu]

Nice to see the shutdown-agents workaround.

Do you experience the same issue with clojure -m benchmark.core?

[andy.fingerhut]

I do not want to discourage you from updating ClojureDocs with useful info, but the issue you came across has been documented there for years now. See the 4th example for clojure.java.shell/sh here, which refers to the page on future for more details: https://clojuredocs.org/clojure.java.shell/sh

I suppose the full details could be copied to the 3 or 4 functions that are affected – when I added those notes it seemed best to have the full details on one of them, and cross-reference from the others.

[didibus]

The use of shutdown-agents isn’t a workaround, but the intended normal use. The issue here is sh doc-string not mentioning its usage of the agent pool.

Even better (and a good habit), to call System/exit in your scripts when you want to exit. It should also cause the agent pool to be shutdown immediately.

[geokon-gh]

Oh yikes, I didn’t actually see your code comment @andy.fingerhut . I’ll add an extra entry in the Notes section so it’s more prominent. And yeah @didibus this to me is a problem with the doc string really. Thank for the pointer to System/exit

The only inconvenience here is that adding either System/exit or shutdown-agents makes it so I can’t run (-main) on the REPL b/c it kills the whole environment hahaha. The solution I guess is just not to run (-main)

I’ll keep (-main) as a non-REPL entry point for the application and have it as a simple launch wrapper that runs your applications and then kills the environment

Let me know if I missed anything guys. Appreciate the help :)))

[geokon-gh]

I do want to remark that the clojure.java.shell/sh in general is a weird one…

For instance (maybe a topic for a separate thread?)

(println (:err (sh "time" "ls")))
Gives me
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 2372maxresident)k 0inputs+0outputs (0major+114minor)pagefaults 0swaps

While in xterm/bash I see something a little cleaner

$ time ls > /dev/null

real    0m0.004s
user    0m0.001s
sys     0m0.003s

This might be some other undocumented magic? Or am I doing something wrong?
(I’m just trying to make a simple benchmarking script that runs an app with different params :S)

[pesterhazy]

You may be seeing the difference between /usr/bin/time and the the bash time builtin

[geokon-gh]

That’s a great guess
But I just tried running (sh "which" "time") and I get the same /usr/bin/time as when I run in the terminal

However, I accidentally did find that I could replicate the weird time output on the terminal. I needed to modify the an environment variable to run a different program. So I was running something like

LD_LIBRARY_PATH=. time MyTestProgram

and then the output was funky just like in Clojure. So it’s a weirdness with time and not Clojure’s sh

EDIT : I see what you mean now about the built in time. Seems it’s getting called in the case with the prettier output. Details for anyone curious: https://askubuntu.com/questions/434289/why-doesnt-the-time-command-work-with-any-option

[pdbrown]

Hey @geokon-gh, I ran into this same issue recently (scripting with clojure.java.shell too), and found another way to ensure the jvm exits promptly by using set-agent-send-executor! and set-agent-send-off-executor! to install daemon threadpool factories. It’s a little more code, but the nice thing is that I no longer have to ensure all possible exit points call (shutdown-agents). This goes near the top of a utility lib I use in my scripts:

;; Configure agent threadpools to use daemon threads so we don't have to
;; remember to call (shutdown-agents):

(import '[java.util.concurrent Executors ThreadFactory])

(defn- daemon-thread-factory [name-format]
  (let [counter (atom 0)]
    (reify ThreadFactory
      (newThread [_ runnable]
        (doto (Thread. runnable)
          (.setName (format name-format (swap! counter inc)))
          (.setDaemon true))))))

(shutdown-agents) ;; stop the default threadpools before installing our own

(set-agent-send-executor!
 (Executors/newFixedThreadPool
  (+ 2 (.. Runtime getRuntime availableProcessors))
  (daemon-thread-factory "clojure-agent-send-daemon-pool-%d")))

(set-agent-send-off-executor!
 (Executors/newCachedThreadPool
  (daemon-thread-factory "clojure-agent-send-off-daemon-pool-%d")))