Interesting point. I don’t know of any related material, like a study of methodologies or practices that really look at the impact of just putting a lot of thought into a project before you get started.
I think, part of it depends what you take HDD to mean. I never took it to mean waterfall, which I believe has some empirical data showing that it doesn’t really improve on deliverables, quality or timing.
I always took HDD to mean, don’t just pick the first solution that gets you to the next move. Spend more time exploring the solution space, and think a few moves in advance (or as many moves as you have time and patience), to really figure out the better move to make.
Which, hey, maybe it’s a waste of time, if the move you end up taking after doing the HDD on it isn’t any better then the first move you thought of making for example. Or it’s only marginally better and not worth the extra time in the end.
But, I also always thought of the Rich Hickey talk as more of a thought experiment. Like what if you did take way way longer to explore the solution space at every move? What would happen? And that this is what he did with Clojure, which you may argue did result in something better or not.
I think for me, the biggest contrast to HDD is what I call Fast Software (a play on fast fashion).
The idea is, build software as quickly as possible. Assume you don’t care for it to be throwaway. That’s the whole point. Isn’t good enough? Throw it away. Doesn’t work well? Throw it away. Can’t extend to your next use case? Throw it away. Build a whole new one super fast again. Rinse and repeat. And maybe in the same amount of time as a slow and well thought out approach like HDD, you end up with a result which is just as good or better.
A benefit of fast software as well is, the business gets value much quicker, even if the first iteration wasn’t very good, it often can already be usable.
A downside of fast software is that once people start using the first iteration, transitioning them to the second, third, forth, etc. becomes a ton of work and sometimes never happen. So you don’t get to go through the process more than once.
Personally, I think for learning and honing your craft, fast software is pretty good. Want to be a better programmer? Write a bunch more programs, and a much more varied set, always pushing yourself. I think Rich Hickey talks about this as well.
Anyways, I have no conclusions here, just more thoughts.