Mind my triple-post but YES! 100% Going from “I’ll save this file, and wait for the browser to refresh so I can click the button to open the modal, click edit, add an item to the table, and click save to see if anything will show up in the console” to “I’m just going to eval this function real quick and see what it outputs” is quite the brain exploder.
According to the code posted and @eccentric_j 's comments, I took it to mean that the function can be called as oauth.authorize(odata,token) (as shown in his code) and oauth.authorize(odata) without token which would have been correct in this case because the “token was optional” in some calls.
And the later agreement that code assist showing multiple arities would have made that clear in code assist seems to support that reading.
So the problem here in this concrete example is that JS functions do not need their declared parameters to match the actual arguments they will accept (indeed, you can declare a JS function with no parameters and call it with any arguments you want). Which means there’s nothing to offer code assist, to help with this problem.
Well, if that’s the case, this is a JavaScript issue that doesn’t really plague Clojure and doesn’t require static type checking. Though you will have this same problem when doing interop in ClojureScript.
Specifically, this is about correct arity checking. Not something you need types for. JavaScript is notorious for lacking more than just static type checking, this is one of them.
- When calling a ClojureScript function with the wrong arity, you get a compiler warning.
- When calling a Clojure function with the wrong arity, you get a runtime error (maybe compiler error as well can’t remember).
I wonder what the JavaScript world does for this? Do they have linters? Are their editors able to show the possible arities with static analysis? Or would just showing the method doc help? Does JS even have docs? Or is doc just comments in the code that get bundled out?
Also I wonder if at runtime you can ask the JS method for its possible arguments? Not sure?
But I am also confused, because if you can call it as such:
oauth.authorize(odata,token)
or
oauth.authorize(odata)
Why did it not work when their code were calling it with the former correct 2-arity of passing in odata first and token last? Is it that the API actually needs you to do some checks to figure out if this is a case where a token should be provided Vs a token shouldn’t be provided? And in the case it shouldn’t, then passing in nil as the token or empty collection isn’t valid and will result in unauthorized, and you are supposed to call the 1-ary variant?
@eccentric_j clarifications?
- How big a deal is it? 100kLOC or 100loc?
This issue on its own wasn’t so bad but it did hold us up getting some features out by our expected deadline. However, given the nature of this project I’m working on leveraging CLJS like a syntax around JS, these kinds of issues seem to be occurring more frequently.
For instance a coworker found a particularly amusing quirk:
(let [mergee {:test true}
get-array-map-of-size
(fn [n] (->> (repeatedly #(str (random-uuid)))
(take n)
(mapcat (fn [x] [x nil]))
(apply array-map)))
m1 (assoc (get-array-map-of-size 6) "test" false)
m2 (assoc (get-array-map-of-size 7) "test" false)]
(map #(get % "test")
[(-> (merge m1 mergee) clj->js js->clj)
(-> (merge m2 mergee) clj->js js->clj)]))
;; =>
(true false)
If static types were required, or even if specs were required on JS interop then I think it could have helped point out that it’s optional. If not provided by the library, requiring us by process to learn it to draft the spec.
I’ve noticed flowtype has an ability for devs to pull in type defs for existing libs? Is there a way to do that with spec? At the very least we might be able to push something to help others working with that library from encountering that error because at least there’s something concrete to look at to show its nature and mark that argument as optional. Not exactly a silver bullet, but could be helpful for more complex JS projects.
The issue with that token argument is that it’s only factored into the signing so it didn’t output anything seemingly different from the working headers.
The library in question is GitHub - ddo/oauth-1.0a: OAuth 1.0a Request Authorization for Node and Browser.
When it’s working for us the token looks like:
{"Authorization" "OAuth oauth_consumer_key=\"xxxxxx\", oauth_nonce=\"0uK2CjB6B5kCFoMFY4ccBT7v9BfEwmLK\", oauth_signature=\"RnZQvn6PTuh2gmV5EOA8%2Bo1wVJM%3D\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1587952561\", oauth_version=\"1.0\""}
When it’s not working it looks like:
{"Authorization" "OAuth oauth_consumer_key=\"xxxxxx\", oauth_nonce=\"hS2VyZdHDxT9XW4pM9Rh0QlMaS5ZXQXp\", oauth_signature=\"JWrOZmRbXcMJgeOEUbyCwFtJEuo%3D\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1587952355\", oauth_token=\"xxxxxx\", oauth_version=\"1.0\""}
I get that this isn’t something Clojure can provide a silver-bullet for, but I think writing and publishing a spec for libs on our part could be helpful as a better reference than an inline comment in a code example below the fold 
That’s a really complicated way to show that conversion back and forth between Clojure and JS is destructive 
=> (js->clj (clj->js {:im-a-keyword 10}))
{"im-a-keyword" 10}
But if you read the doc for it, it says:
With option
:keywordize-keys truewill convert object fields from strings to keywords.
(js->clj (clj->js {:im-a-keyword 10}) :keywordize-keys true)
{:im-a-keyword 10}
But still you have to be careful, because you can’t have it both ways, this is still destructive. Either string keys become keywords or they remain as strings. So if you run your colleague’s example again, it still doesn’t work:
(let [mergee {:test true}
get-array-map-of-size
(fn [n] (->> (repeatedly #(str (random-uuid)))
(take n)
(mapcat (fn [x] [x nil]))
(apply array-map)))
m1 (assoc (get-array-map-of-size 6) "test" false)
m2 (assoc (get-array-map-of-size 7) "test" false)]
(map #(get % "test")
[(-> (merge m1 mergee) clj->js (js->clj :keywordize-keys true))
(-> (merge m2 mergee) clj->js (js->clj :keywordize-keys true))]))
;; =>
(nil nil)
Because “test” has becomes :test.
I don’t know if its fair to call it a quirk, it is something to be aware off, and something that can cause bugs for sure. But I don’t know that there would be a good solution to this. JavaScript isn’t powerful enough to encode the types of Clojure, that’s why EDN is superior to JSON But also, you need to hand over something that is standard JS, so if you went with a custom encoding, it wouldn’t help with interop, that’s why clj->js and js->clj choses to be lossy.
Well, this is what core.typed was doing, but everyone felt it too painful to retrofit types in all kind of things without types.
Also, if I understood the issue, then I agree with @seancorfield, types would not have caught this error, since both calls are valid and type check. The issue is your particular auth provider somehow was not compatible with both arities of the library. Don’t think there’s an easy way to solve that, they either need to follow the spec, or have better doc.
That’s not the issue. The quirk is not that the keywords become transformed from :key → “key” → :key. The issue is that if you set “key” false and :key true on an PersistantArrayMap (fewer pairs) vs. a PersistantHashMap (more than 6 pairs) it may read a different value based on the order of the pairs in the array map.
Agreed, a static type checker would not have caught it. But a spec or a static type hint could have clued us in sooner that it’s an optional argument. Now circumstantial, this is entirely our fault for not reading the docs more carefully. However, this does represent a larger problem when interoping with JS that I think can be improved on the CLJS side.
It’s a tool to pull type defs for third-party libs. I’m curious if we could set something up like that for spec? That could help a lot in taming the wilderness that is JS. Maybe even add some tooling to transform a spec into something more concise and readable for a quick sense of what’s expected?
Hum, I’m not sure that’s correct. I mean, obviously maps don’t guarantee order. But the issue in your case is that the conversion to JS creates two equal keys. I might be wrong, but that’s what it looks like to me.
You can see that because the discrepancy in results isn’t there if you remove the conversion to JS and back:
(let [mergee {:test true}
get-array-map-of-size
(fn [n] (->> (repeatedly #(str (random-uuid)))
(take n)
(mapcat (fn [x] [x nil]))
(apply array-map)))
m1 (assoc (get-array-map-of-size 6) "test" false)
m2 (assoc (get-array-map-of-size 7) "test" false)
(map #(get % "test")
[(-> (merge m1 mergee))
(-> (merge m2 mergee))]))
;;=>
(false false)
Sorry for the formating, I’m on my phone right now 
I guess I understand what you mean in that, when clj->js is called it will walk over the map entries in a different order and that will determine which of the two equal “test” keys get inserted last in the resulting JS map. But this only matters because the keys both convert to the same key.
I think some other languages (namely Python) seem to always guarantee insertion order on maps, but none of the languages I’ve ever used did, so I’m not really surprised when order dependent logic on maps doesn’t give consistent results. I’d still contend the biggest issue in this piece of code is that the conversion to JS is lossy and it forces only one value to win over the other, since through the conversion they now share the same key, where in ClojureScript they were just distinct entries.
Just to reiterate, they don’t get transformed as such. It goes: :key -> "key" -> "key". That is, at the end of the round-trip, your keyword has become a string, even in ClojureScript. And if you chose keywordize-key true then the other key goes: "key" -> "key" -> :key. So in both situation the key will become same, either both string or both keyword.
I’m not so sure. An optional argument doesn’t mean that having it will fail authorization. I mean, what if you did see it specified as:
authorize(odata : Odata, token : Optional<Token>) : String
The intuitive reading would be, oh, well ya, it takes an Odata and a Token and we are passing both, so why isn’t it working, we’re clearly calling it properly.
Now maybe a Spec would do, since it could specify something like:
(s/def authorize
(s/or :provider-x (s/cat :odata ::odata :token ::token)
:provider-y (s/cat :odata ::odata))
Or something like that, but that assumes that providers are a known thing. I don’t know if that’s the case here?
My perspective is documentation would be best here. The provider should have had better documentation. And ideally, the provider shouldn’t have an API that isn’t compatible with the method signature of the library.
I think this brings up a good point, I’ve faced issues like this before. When you call a remote API, there is no possible static type checking, you are crossing the app boundary, all bets are off. Even if the API gave you a typed client or a fully detailed spec of their API, they might be inaccurate, and then you’re kind of screwed, because they are lying to you. It just becomes really hard to figure out, it’s like having a bug in your language compiler, which has happened to me before as well.
I agree with this, though I don’t think your given example falls under the things that we could solve here. But for other kind of issues, like if you passed in 3 arguments to a method that only takes 2, or if you pass in A, B, when the method wants the order to be B, A, etc. That could be made better with tooling.
I also want to say something a bit more general. I come from a background of static typing like I previously said: Scala, Java, C#, ActionScript 3 and C++. That’s been my professional experience prior to Clojure. And dynamic langs I had use before but not for work were Python, JS and PHP.
I have noticed a lot of devs with the opposite experience, they come from Python or JS backgrounds, thinking static types will solve all their problems. Often times, the examples they bring up, from my experience, arn’t solved. This is for two reasons I’ve observed:
-
They wrongly think static types would catch their specific scenario where it wouldn’t. Or it would, but only in the most rigourus typed language like Idris, Ada, or to a lesser extent Haskell, sometimes Kotlin. And only if the prior programmers put in the work to properly specify the bounds strictly enough in the first place.
-
Most of the “big” WTF moment happen either at app boundaries where there are no more types, or what often happens is static languages retrofit typeless dynamic programming into them in an ad-hoc bespoke manner. So you end up with code that uses strings, logic that’s embedded in config files, things typed as Any or Object, etc. And for both of these situations, since the language assumes everything will be properly statically typed, when you face such a scenario, it is even harder to debug, because you least expect it, and the language has no tools and no best practice to deal with it.
So that’s something to ponder about as well.
Okay, but, having said all of that. I am super into improving Clojure tooling. With my anakondo plugin for Emacs I’m tackling Clojure for now, and I just added Java auto-completion, and I wanted to add method and field completion for just this kind of thing. It’s nice when you interop if the editor can list you the available methods, their arities, and their param names, and maybe even the doc for it all inline. I’m still working on adding it for Java. I don’t know if I could do it for JS, but I’ll be looking at ClojureScript eventually. So I’m open for ideas.
For example, that flow repo might be something I can leverage. I could see if I can look up the method and library from it and pull down info to show inline on the editor. That might be possible. It wouldn’t type check, but it shows you the type signature from the flow repo for the method being called.
I think this is a fascinating thread but it feels like it’s gone off into the weeds of a “static vs dynamic” typing discussion which really isn’t relevant to Clojure as far as I’m concerned. Clojure is a dynamically typed language. Some people prefer that, some people prefer statically typed languages. Sure, source code analysis tools (linters) can identify potential problems in your code but can also flag false positives (that was my biggest problem with core.typed each time I tried to use it). I used to work for a source code analysis company back in the early '90s – we wrote tools to analyze FORTRAN, C, and C++ in great depth (and, after I left, the company added a Java analyzer). There’s a lot you can do with data flow analysis but a) you have to have some really smart and deep analysis and b) you are either going to miss real bugs or you’re going to flag false positives. Static type systems are in the same boat: if they’re too basic, they don’t catch a lot of bugs (Java’s type system doesn’t help with NPEs or some cast exceptions), and if they’re really strict, they disallow programs that would otherwise be valid at runtime (Haskell, for example, prohibits a lot of code that would be valid without types at runtime – and as the whole transducer debacle showed, something that is easy to express in Clojure is almost impossible to express correctly in Haskell without degrading to Any as a base type). That’s the fundamental flaw in the argument that static typing is a “superset” of dynamic typing with everything being Any/Object: you end up with an untyped system which is not what dynamically typed languages provide: they are typed, but at runtime.
The REPL workflow aspect is much more important because we all seem to agree it is central to Clojure but we also seem to agree that a lot of people coming to Clojure don’t seem to be able to develop/learn a truly effective REPL workflow.
Perhaps at this point it is worth breaking this thread off into two subthreads? One that can continue the static typing discussion for those who are interested, and one for the REPL workflow?
1 Like
FWIW I use Clojure in fairly exotic applications. Not only Java interop, but also native interop and even GPU interop. And I too run REPL for days and weeks. I usually have to restart it due to system update (Arch Linux) or due to my Emacs update or Nvidia GPU driver, and rarely due to state of my Clojure app in the REPL, since even if you do mess up the state of the program (which is hard to do if you follow FP) you can simply reload one or more namespaces in your program and keep using your REPL session for a long time…
4 Likes
Oof do you hear that? That’s the sound of my theory sinking into the depths because of the gaping hole your experience torpedo just punched through its hull.
That said, it sounds like in some cases a dirty REPL state is possible but there’s no relationship or even a correlation between project type and REPL experience.
2 Likes
After talking to Sean a bit more via Slack I can see where I didn’t represent my issue very well and as we explored it more it’s not a cut and dry example of where automated tooling could help.
However, the conversation did help me refine some more specific topics\questions I think would be interesting to discuss in the coming weeks.
For the record, I’m not advocating for static typing or changes to Clojure’s core. The relationship between the contact’s insight and my own experience is that I can see where static typing can give you more certainty just by reading the code, or through tooling around type hints. There are of course exceptions and limitations of static typing too based on the experiences didbus and others have mentioned but I can see value in the certainty you get where it works.
In an attempt to steer this discussion back on track:
What can we do to give contributors\teammates more certainty about what our code does without static types? Are there any examples of a repo that you found really approachable or maybe one where you think it could be improved?
For example, I try to denote relevant keys in my docstrings and a hint about the return type:
(defn paginate
"
Records are paged by 50 items so we want to fetch them all
Takes the following:
- a response hash-map with total_pages and results string keys
- an API consumer key string
- an API consumer secret string
Returns a highland lazy stream of all leads
"
[response key secret]
(let [response (js->clj response)
total-pages (get response :total_pages)
records (get records-response :results)]
(-> (range 2 total-pages)
(stream)
(.flatMap #(stream (p/-> (records/fetch key secret %)
(get :results))))
(.reduce records concat)
(.flatMap clj-stream))))
Personally, I’ve soured on structured docstrings. I feel everyone usually gets to the point where they just try to satisfy the requirement instead of writing meaningful hints so I’d rather see a little content that’s meaningful than like data {object} - The data to operate on.
1 Like
Well, currently, I would say those are the options:
- Provide a Spec
- Have a good doc string that documents the shape and structure of each arguments and return value, and the possible set of semantics.
- Use destructuring, but don’t use
:ason it. That means, be explicit about all the keys or positions (if a sequence) that you make use of. If you define an:asit means the function is generic over all keys/positions and makes use of them all. - Provide unit tests that also act as a walk-through of the how to use the function. A literate style is good here.
Maybe even more so I would say:
- Try to write as many pure functions as possible
- Make sure impure functions that are not entry points only do side-effect, nothing else, and make sure they are never called from anywhere but the entry points themselves.
- Keep entry points only handling data flow, and move all side effects to functions that only do side-effect, and all business logic to pure functions.
Looking at your example I’d also add:
- Work fully in ClojureScript, only convert to and form JS at the boundaries.
- Define domain entities, if I see multiple places that take a
responseI’d expect that they all take the sameresponseentity. If not, I’d expect them to have different names, like,paginated-response,customer-retrieval-response, etc. This doesn’t hold for generic functions, like if I seekandv, I assume to function operates over any key or value in a generic way.
Those are the tips/tricks I know of for now.
I’ll take a crack at this one, since I’m one of those people for whom using the REPL has been a completely “meh” experience.
I agree with what people have said about it being a switch in mind set. Using the REPL seems to be about figuring out how to get to the next step from where you’re currently at. I.e., I’m at A, how do I get to B?
The way I code is more along the lines of, “I’m at B. That’s bad. How did I get here and how do I make sure that doesn’t happen again?” Maybe I can get to B from A, E, or H. I can write tests that set things up to start at A, E, or H and then test that B isn’t reachable from there. Once those are written, they can be added to a regression suite to make sure that future code changes don’t cause B to happen again.
With the REPL, the only way to rewind it is to restart it. Once you get to B, the only way back to A is a restart.
To the point about the REPL state, I lost count of the number of times something worked in the REPL but failed when I restarted. I got to where I don’t trust the code in the REPL, especially if it’s been running for a while, and need to run the code from scratch, anyway, to guarantee it works.
For me, the best development process has been rapid, test driven development. I use VSCode and will usually have at least half a dozen or so tasks set up to run whatever tests work the code I’m currently working on. That flow, for me, is comparable to the REPL in speed, doesn’t have the gotchas that the REPL does, and works regardless of what language I’m using, be it C, PHP, Clojure, Javascript, Common Lisp, etc.
Not saying the REPL is bad, just that it’s not for me.
1 Like
Thanks for taking a crack at it 
, but it still baffles me. I guess I did quickly learn to just know why certain things don’t work when not at the REPL, that’s mostly because the order in your source files isn’t always the order you send forms to the REPL, and a few other things are due to some magic that certain REPLs put in place which can backfire. I admit that’s some of the things to learn, but for me, it’s like learning to drive a car, still worth it compared to walking everywhere.
I also don’t have this issue. Rewinding is just resetting the state back to what you had before. Doesn’t mean you need to restart the whole REPL. And I tend to write mostly pure functional code, so I don’t often deal with state like that.
But, I think I remember you talking about your test workflow, and it sounded quite elaborate and very quick, something I thought I should try myself as well. One thing that maybe I’m curious to know, you don’t run your tests in the REPL either then? You start a separate app process for each one? Cause even my tests run in the REPL, that’s what the Cider test runner does. I believe most test runner I know also actually don’t isolate tests one from another, so I’m curious, if that’s the case for you as well, why don’t you get lost within your tests when they similarly run all together in the same process with the effects of one possibly messing up the other? I’m guessing it’s because you have good setup and teardown around them, why can’t you perform the same setup and teardown at the REPL?
Honestly, I’d expect your reaction to it to be the same as my reaction to the REPL. “When it works, it’s about the same as what I’ve been doing. When it doesn’t work, it’s annoying. So, I’ll just stick to what I’ve been doing.” 
Right, separate process for each test with setup and teardown. You could do the same thing with the REPL. I tried it that way for a while and slowly drifted away from it.
At the end of the day, I think both approaches are rapid development techniques and achieve similar goals. Each one may have features that the other doesn’t, but it’s one of those things where you use the one that works for you. If viewed that way, I definitely agree with you about not understanding how people don’t do it.
I remember in college, where the core language was C, and I was struggling with one piece of an assignment. I went into the lab to work on it and asked one of my classmates how they were doing on it. They said they were completely done with it. I asked about the part I was struggling with and they said they got that done, too. I asked them to run it for me so I could see it was possible and I was just missing something. Their response to me asking them to run it? “Well, I haven’t compiled it, yet.” I don’t get how people can code that way.
1 Like
Ya, I like the exploratory nature of a REPL and wouldn’t want to give that up. But I also have tests. Now I mean, if writing a test and running the test basically entails almost the same user experience, we can probably consider them same.
So, one thing already that I think is much similar between the two is that when I say REPL in Clojure, I don’t mean the command line input based one. I mean pressing Ctrl+Enter anywhere in my source code and having the form at cursor executed and returning its result.
This means I’ll have commented forms either with comment or #_ that perform setup and teardown of certain resources for when I want to be able to run any form in the code and have their necessary environment or inputs available to me.
Once my function starts to take shape in terms of what I really want it to do (generally happens after some level of REPL based exploration), I might open up the corresponding test file and start writing test cases there. Though it depends, because in my REPL I often go the extra mile and setup full integration state, and in my tests I might restrict that more to unit tests, or if I think necessary I might go out of my way and have an integ test as well, but the work involved to stand up state for an integ test often is more than in the REPL, so I don’t always.
At this point, my tests are still in my REPL even though they are also inside my test file. Because my files and my REPL have the same UX. So I’ll be re-running my tests through the REPL still.
So that’s my workflow. But, if instead of the REPL, you could very quickly start and setup/teardown your tests. I guess I could in theory use them for exploration as well. I mean, that’s what I used to do in other languages. Except starting and running tests in the languages I’m used too, Java, Scala, C#, C++ is nothing that’s easy or quick. But if you were in a place where the only difference to what I described above is that you are restricted to Ctrl+Enter being allowed only in the test files, and where instead of running the form at point, it ran the test at point, and the speed of bootstrapping is just as fast, ya, that gets real close to having the same user experience I think. With the benefit like you said of isolated state and always starting things from a blank slate so the last execution can’t affect the next one. So there’d be that. Though sometimes I choose to keep the last execution state around on purpose, with tests I’d have to do a bit more copy/pasting to carry over the setup of one in the other and all, but I guess that’s manageable.
Where I think REPL driven development goes even beyond this though, is something that in all honesty Clojure also doesn’t fully realize. But if you ever try and code Emacs you get the full beauty of the idea. Being able to modify the running production program in real time. Hot Swapping logic in and out, have a bug? Just fix it. Don’t even need to restart and you fixed the issue + you saved the fix back to the program source file so when you restart the fix will still be there. Now for big enterprise service and all, that might be playing too fast and too loose. But from a usability as a programmer, man, if every app were like Emacs (or if Emacs had become the Linux desktop manager as it was envisioned and all desktop apps had been Emacs apps) that would have been an awesome (maybe except for performance) world.
I do use REPLs in production as well, mostly for debugging, and the source of the app on a server isn’t under git control, so it’s not exactly the same flow. So it’s more of a convenient debugging tool for being able to run certain things and inspect state in prod.
Yeah, the REPL does have some nice features. I agree that talking about the REPL assumes editor integration using SLIME, Calva, etc. I do understand why people like to use it.
Back to one of your original questions about why other languages aren’t heading that way, it’s because it’s scratching an itch that isn’t really there. The feedback loop isn’t an issue in any language. The worst I’ve experienced would be C, C++, or Java. A good build system mitigates it, though. The reasons I choose not to use those languages have nothing to do with overly long compile times. If I listed the pain points when coding in them, I doubt that would even crack the top 10. I know this falls on deaf ears talking to Lispers, but using the REPL as a key selling point won’t attract many buyers.
I actually disagree about Emacs. I used it exclusively for the better part of a decade, so am quite comfortable with it. I started moving away from it when Java took over and the Java IDEs are simply better to use than Emacs for working with Java. Then I managed to lose my init files and tried to resort to plain, out of the box Emacs. I forgot how unusable I find that to be. I managed to reconstruct some of my init files and still used it here and there until VSCode showed up. At this point, I never use Emacs anymore.