How to send http requests the curl way? or: help with http-clj

[magnus0re]

I just want to convert CURL examples to clojure code.
I’m trying to send

curl -X POST --header "Content-Type: application/x-www-form-urlencoded" -d "client_id=MYCLIENT_ID&scope=api&client_secret=SECRET&grant_type=client_credentials" API_URL

with clj-http and I’m really struggling

So far, I’ve tried things like this:

(require '[clj-http.client :as client])

(client/post  (:authorize-uri client-params)
                {:save-request? true
                 :debug true :debug-body false
                 :headers
                 {:content-type "application/x-www-form-urlencoded"
                  "-d"
                  (str
                   "client_id="
                   (:client-id client-params)
                   "&scope="
                   (:scope client-params)
                   "&client_secret="
                   (:client-secret client-params)
                   "&grant_type="
                   (:grant-type client-params))}})

But honestly, I just want curl mechanics, make the damn string, and don’t care about obtuse nested maps… Every example out there is curl, So I want to speak curl as well in my programs. building strings from data is easy enough, I don’t need a library to do that for me, It only becomes more cognitive overhead with no added benefit. (sorry for the rant)

I keep getting failures from the server when sending it from clojure, and from the debug info it seems that there is additional data, like “accept-encoding” “gzip, deflate” injected into the header by some middleware in the library.

Any help is really appreciated.

[borkdude]

Perhaps you can use GitHub - babashka/babashka.curl: A tiny curl wrapper via idiomatic Clojure, inspired by clj-http, Ring and friends.. With the :debug setting it will return the command string it used to call curl. You can also pass raw arguments with :raw-args.

[kopos]

You can use :query-params. Also note that save-request, debug, debug-body are not needed

(require '[clj-http.client :as http])

(http/post "http://xys.zom"
  {:save-request? true,
   :debug true,
   :headers {:content-type "application/x-www-form-urlencoded"}
   :query-params {:client_id "234"
                  :scope "realm"
                  :client_secret "asd"}})

[camdez]

@magnus0re

I keep getting failures from the server when sending it from clojure, and from the debug info it seems that there is additional data, like “accept-encoding” “gzip, deflate” injected into the header by some middleware in the library.

You are entirely right that clj-http injects some additional functionality via middleware. It is, IMHO, a pretty sensible set of default middleware, so I worry a bit about how the server is implemented if these things are causing failures, but I understand the need to focus on the problem at hand / cope with the server you have.

If you want to disable the default middleware, that’s as easy as:

(client/with-middleware []
  (client/request ,,,))

However, I would be mindful of this from the with-middleware doc string:

  It is highly recommended to at least include:
  clj-http.client/wrap-url
  clj-http.client/wrap-method

  Unless you really know what you are doing.

Cheers.

[camdez]

Oh, and if you really want to get closer to curl, just do something like this:

(ns foo
  (:require [clj-http.client :as client]
            [clj-http.headers :as headers]))

(defn request [req]
  (client/with-middleware [headers/wrap-header-map
                           client/wrap-query-params
                           client/wrap-url
                           client/wrap-output-coercion
                           client/wrap-method]
    (client/request req)))

;; `:keys` here are just included for documentation
(defn curl [method url & {:keys [headers query-params] :as req}]
  (request (merge req {:method method :url url})))

;; curl -X POST --header "Content-Type: application/json" -d "name=Bob" "http://google.com"
(curl :post "http://google.com" :headers {"Content-Type" "application/json"} :query-params {"name" "Bob"})

Almost the same length, no nested maps, and easier to edit / maintain / enhance, in my opinion.

[magnus0re]

I had totally given up on this, I just used curl from the shell…
The original post was kind of fired in anger, so I regret the wording in that a bit, but this worked so nicely!
Yes, the server is probably just made on a shoelace budget, it doesn’t even give proper error messages.
Thanks a lot!

I feel that this library has a lot of power, but I don’t really know how to leverage it.

[camdez]

@magnus0re

The original post was kind of fired in anger, so I regret the wording in that a bit,

Heh, no worries. I spent literally days trying to get things to work in Scala that I could have done in five minutes in Clojure, so I know how that kind of things feels.

@magnus0re

I feel that this library has a lot of power, but I don’t really know how to leverage it.

Yeah, I think it’s a really good library. I usually follow the basic approach I outlined here of making my own wrapper around request/client that adds functionality I want to use across a series of calls. For instance, if you’re working with a particular API, there’s probably a common set of things like content-types, authentication headers / params, or parsing that you can encapsulate in your own little request lib.

There are some nice little thing it provides that are not immediately obvious like :oauth-token, which gets transformed into an appropriate OAuth header. That actually shows the way to more advanced stuff: add your own new concepts to the request map and add your own middleware which uses it to construct the exact request you want.