Announcing hotload: Code Python 'like it's Clojure!

Disclaimer: We won’t be able to match Clojure’s development experience. But we’re getting there!

So! We all love the REPL. It makes it possible to work on pieces so small that errors are trivial. Heard of Kent Beck’s test && commit || revert? Write code. If the test fails, revert your commit. What does that do to your development flow? It makes you code in smaller increments. What’s another good way to ensure that you write code in small increments? Use the REPL! Some things I like about REPL-driven development:

  • Reloads that are so fast in speed that you don’t notice their delay.
  • State persistence between reloads. I can define something off in the background and query it later.
  • A single key press to get a new result.

The last year, I’ve been writing a lot of Python. And I’ve been playing around with how good a Python workflow can get. Turns out, that’s quite far! And now I’m releasing that effort.

hotload watches your files and reloads your Python modules in-place. Please check it out! I’d love comments and feedback.

3 Likes

I’m happy you’re trying to get back some of what Clojure brings while using Python. Hope that helps fight the itch :yum: I’ll keep it bookmarked in case I’m ever forced back to Python.

Just out of curiosity, is data-science your domain? Or did you face challenges making headway convincing coworkers to adopt Clojure over Python?

Hello!

Let my try to give an answer! I don’t have a typical computer science background, but applied data science is a fitting label from a CS point of view. My master’s degree is in computational mechanics – which is how civil engineers do the math to ensure that bridges and other constructions can take the load they should (“finite element analysis” in civil engineering lingo). Under the hood, computational mechanics is a set of equilibrium equations, and data visualization tools that you use to interpret the results (which are often more than 2 GB of data). We have access to quite good products for doing this on the market, which often support both point-and-click to get what you want and some kind of scripting interface.

Lately, I’ve been working at a startup with other people from the civil engineering industry. We have blended the traditional project-deliver-PDFs with some lessons about continuous delivery. For instance we prefer to visualize our current-best results on a web dashboard in the project, to enable better communication.

So where does that leave us? Since we’re not a team of dedicated software engineers, Clojure is a real investment. In addition, commercial software like Abaqus has a Python API, which sets its own requirements! Regardless, I’m toying with this idea that we should all just bite the bullet and do our software-enabled work with Clojure – since Clojure is very well suited to the kind of interactive, improvisation-based work we do a lot of.

If I may ask, what has been your path? I’ve kind-of been guessing that you have a longer experience that involves developing backend systems, and perhaps some Java experience?

Teodor

2 Likes

Ah ya, that makes sense. Honestly, it does seem Python might be more fitted for that. Having readily available libraries and tooling integration for practical work projects is a huge win. In those scenarios, going with the path of least resistance is often the right thing. Even though I’m a big Clojure advocate, I’m a pragmatic at heart. For those use cases, even if Clojure had it all, I don’t even think it would improve that much the experience over Python. So all the power to you :facepunch:

Your hunch is correct. I’m your typical big backend enterprise software engineer. I mostly work on distributed systems, backend services, and the occasional front end generally for internal use only. My background is C++, Java, C#, ActionScript 3, Python, JavaScript mostly. My work is a perfect match for Clojure’s strength. Big ETLs, large scale data processing, backend services, SPAs, cron jobs, daemon processes, all those kind of infrastructure-ish level software projects are the ideal match for Clojure. So I’m lucky in that sense.

Keep us posted on how things evolve for you. I think there’s a lot of work happening by motivated individuals trying to prep up Clojure/Script to compete more seriously with Python in the scientific computing and data analysis, visualization and exploration space. So hopefully it can eventually become a more appropriate target language for those domain. In the meantime, I think it’s cool how you’re trying to use Python more interactively.

2 Likes

Hello!

I just completed a (tiny) CLI for hotload. You can now use it like this to watch all your python files without restarting the Python interpreter:

find . -type f -name '*.py' | hotload init.py

This will reload init.py when there are any changes to Python files in the current directory. For me, this meant that I’m now running all my Python code with hotload as I’m developing it, as the usage is simple to remember, and I can install it once on each machine.

Check out the readme for updated installation instructions:

As always, feedback and comments are much welcome!

Teodor

Python is like COBOL, FORTRAN, or C. For a specific time frame, they were the default language. C and C++ sucked big time for web development. It gave us things like Ruby. Python was develop a single language that solved some of the problem. It’s no used in MatLab, R, other sciences. Used in anything that needs a UI, batch jobs. The new perl.

Large companies can buy hundreds of mediocre Python programmers for a few cents a day. It’s so endemic that MIT gave up and is now using it to introduce computing theory. It’s good enough that a company can chose it an not deal with Java/C++ wars.

Hello, @tyohDeveloper,

Thanks for your interest. Yes, Python sure can do a lot of different tasks. I’m finding that when I can’t use Clojure, Python often fits the job. Not that I haven’t seen Python codebases with a mutability soup that makes everything hard to understand. I’ve found that learning Clojure has significantly improved the way I structure my Python code. But I’ve missed quick reloads, and the ability to persist some state across reloads. Which is what I’m addressing here!

I’d love comments on hotload if you’ve tried it out.

Teodor

1 Like

Me too. Python is a lot like perl. You can do anything you want in it. Which can make it unmaintable. Even the first time it is developed on a team. For the mediocre programmer (lots), even when they are writing alone, but over several weeks.

It takes discipline to limit oneself to one style of programming. Even there, there is no guarantee that later teams won’t have the same discipline. If we were Haskell programmers or even Scala, the problem would look worse. Monads are an impossible concept in those languages. Python takes those intractable problem and makes it worse.

Clojure (any lisp) can be used in an undisciplined manor. It still requires a much greater level of discipline to use. Which will make for better systems. Unfortunately, the requirement is you need more than mediocre programmers in the first place. For large corporations that offshore a lot of maintenance it makes it harder to find contractors that can pull it off. Good programmers are good programmers and not cheap. They are cheaper in other countries, but not that much cheaper. Experience in functional programming and lisps aren’t common on the ground. That will make them more expensive.

Having said that, good programmers (can) always create better products. That leads to systems that are more robust and need less maintenance. That equation is too close to long-term-thinking. The people that reach senior management generally can’t master that skill. So talent is undervalued. They’d rather pay $million tomorrow than a penny today. Largely because they’ll be gone when catastrophe happens. Look to Fukushima or Deepwater for perfect examples of that.

Clojure can be easier to reach the ideals of the Haskell programmer. The architect and programmers need to understand the basics of category theory to pull that off. A skill most Haskell programs don’t use. They create Monads, Scala has bunches of libraries to that have pre-built structure. It isn’t as robust as using concepts category theory but it’s close enough for them. They are hideously complex. Clojure is much easier to apply the concepts. Programmers of that level aren’t common on the ground. Good programmers are expensive. See the remarks about senior management above.

Clojure plus ClojureScript is better [my opinion] for systems development. Monads are often brittle. The initial development can be elegant. Those 2am wakeup calls often break the model. At that point, the system won’t recover. Clojure patterns are more elastic. Then I’ve been using lisps on and off for four decades and less than a half decade of putzing with Haskell and a year of putzing with Scala (which I’ve decided isn’t worth the time). ClojureScript in particular allows us to get much closer to the advantages of Python that those languages.

MIT gave up on scheme because one can teach the the theory behind programming with Python. It’s too easy to mix and match paradigms which makes is less suited for teaching theory. Lots of kids come into MIT knowing Python and more can use it to get part time jobs helping out on web sites during and immediately after school. So there’s less impedance getting students into the class to begin with. Again, my opinion, but I think MIT would have been wise to start with Clojure/ClojureScript rather than Python. I don’t know if that would have been true at the time they switched away from Scheme. Now that beginning class is more applied than theory, I doubt there is a chance they can go back. I’ve only been using Clojure for awhile. I don’t know what is was like five years ago. The people at MIT would have a clue. I’m just projecting.

I think your comment on how using Clojure is improving your Python is common. Rather than using every programming patterns that are built into Python, you learn to take a functional approach. It’s made you a better programmer. That’s almost always happens.

Clojure doesn’t have the best tooling. Too many people are Emacs users, they don’t really get the point. Most CommonLisp environments are better that way. They are expensive, especially if you add on all of the web progrogramming libraries. I’m spacing the name, but there was a computer & OS that was designed for AI applications in the 80s, that had a great environment. It would have been great if the tooling people had used a good Smalltalk system for awhile. Those had the best environments I’ve used. Python doesn’t have the ability to build a Smalltalk quality system. But Clojure has the opportunity to get a lot better. Debugging sucks in general. It doesn’t not close to Eclipse/Java. It was designed and built by Smalltalk programmers, which might be why. I think that’s part of the reason Python is better at the things you’re missing from Python.

There have been a few recent announcements about tools that are addressing some of the hot loading advantages of Python. I don’t have any experience with them. If you learn more, I’d be interested in hearing from you.

Have you looked into Clara? I have a feeling that could really help bridge the gap.

That reflects my experience with Haskell. It’s totally great for pure operations on an algebraic data types, but then I find something that doesn’t quite fit the types. For example sate, starting and stopping a server and data with where lots of things might be missing.

There might well be other tools for hotloading that are suitable. I’ve generally found them too complex. In Clojure, we prefer to load forms manually. In the same manner, a simple tool can provide more dexterity.

I assume you’re referring to http://www.clara-rules.org/ ? Not sure whether we’re talking about the same thing. I’m trying to bring REPL-driven development to Python, accepting some compromise. How would a Clojure-based rule system help me do that?

Teodor

I was thinking of using jython.org as an interface. I lost my chain of thought. My apologies.

The Clara reference was an afterthought. Having a TMS might help keep track of system state kind of stuff and/or dependencies for when you need to reboot the REPL. Or when you need to keep elements of state from the interactive use of the repl.

1 Like

i think it is really cool that you have a background in engineering!.. (… me too! :-)… )… also i have just had a quick look at your masters thesis… and i am really really impressed!!!.. now correct me if i am wrong… but you did use latex / gnuplot right?.. so i get how you did those really nicely integrated graphs… but how in the world did you create for example figure 4.1 !!!.. ( … ehh??? there should be question marks also… not just !!!.. weird… i mean… after figure 4.1 i have actually typed a combination of question marks but there are not shown somehow… well doesnt matter :slight_smile: )

Thanks! What’s your background? Structural engineering as well?

You’re right about LaTeX for the thesis. Figure 4.1 was made with TikZ. I really enjoyed TikZ. Unfortunately, since then, I havn’t had the chance to use it much. Here’s the source for Figure 4.1:

\begin{figure}[tb]
	\centering
\begin{tikzpicture}
\draw[->] (0,0) -- (0,2) node[anchor=south]{$x_1$};
\draw[->] (0,0) -- (2,0) node[anchor=west]{$x_2$};

\draw[<->] (-1.5,0) -- (-1.5,3) node[anchor=east]{$L$} -- (-1.5,6);

\draw[fill=yellow, fill opacity=0.2] (-1,0) rectangle (1,6);
\draw[ultra thick] (-2,6) -- (2,6);

% Body force
\foreach \x in {-0.4, 0.4}{
	\foreach \y in {0.5, 1.5,...,5.5} {
		\draw[->, color=blue] (\x,\y) -- (\x,\y-0.25);
	}
}

% Edge load
\foreach \x in {-1, -0.5, ..., 1}
	\draw [->, color=blue] (\x,0) -- (\x,-0.4);
\draw[color=blue] (-1,-0.4) -- (1,-0.4);

% Stress
\fill[shift={(3,0)},fill=blue, fill opacity=0.2] (0,0) -- (1,0) -- (2,6) -- (0,6) -- cycle;
\draw[shift={(3,0)}] (0,0) node[anchor=north west] {$\sigma_{11} = \sigma_y$}
 -- (1,0) -- (2,6) -- (0,6) node[anchor=south west] {$\sigma_{11} = 2 \sigma_y$}
  -- cycle;

\end{tikzpicture}
	\caption{Uniaxial tension due to constant body force and end load. Coordinate system, load, boundary conditions and resulting stress field.}
	\label{fig:bar_loaded_by_gravity_problem}
\end{figure}
1 Like

i did not know about tikz!.. very interesting stuff!

i was trained as a mechanical engineer… now i was in the penultimate year that would graduate as Dipl.-Ing. … so i guess some of the details have changed now… but basically here is what i did: https://www.mw.tum.de/en/studies/prospective-students/bachelor-mechanical-engineering/ then https://www.mw.tum.de/en/studies/prospective-students/master-mechanical-engineering/… also my studies were focused on control engineering… :-)…