Wehrmacht, Elon Musk and Unix Software Development

A look at the power of iterative approaches to solving complex problems in software development, economics, engineering, and war.

A topic which has fascinated me for a long time is how different organizations can achieve vastly different outcomes and success despite having similar amount of resources at their disposal. For instance, how could Tesla outcompete huge automakers in making long range electric cars? How could a startup like SpaceX design a rocket such as Falcon 9 outcompeting far more experienced competitors such as Boeing and ULA, Roscosmos and Arianespace with much more experience and deeper pockets?

Why was the Unix operating system so successful compared to the long forgotten Multics, despite having a tiny team and very few resources at their disposal?

Why did Japanese automakers in the 1980s and 1990s outcompete their American counterparts, which had larger markets, more experience and deeper pockets?

I have chosen to frame all these different questions through the Unix software development philosophy of "worse is better" which emphasis the following points:

• Simplicity — The design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface.

• Correctness — The design should be correct in all observable aspects. It is slightly better to be simple than correct.

• Consistency — The design must not be overly inconsistent. Consistency can be sacrificed for simplicity in some cases, but it is better to drop those parts of the design that deal with less common circumstances than to introduce either complexity or inconsistency in the implementation.

• Completeness — The design must cover as many important situations as is practical. All reasonably expected cases should be covered. Completeness can be sacrificed in favor of any other quality.

I will cover a variety of seemingly unrelated topics and show how these can be related to this philosophy called "worse is better".

I will begin with a story about the German army during WW2, the Wehrmacht. Even though the Wehrmacht was in many ways weaker than the French army, it obliterated the French in record time. Time and again, superior allied forces would find themselves routed by numerically inferior German forces.

Many of the same lessons are, in fact, observed today in the war in Ukraine. Ukrainian forces are beating back numerically superior Russian forces. It is not merely a question of soldiers but also a Russian armed force equipped with far more artillery, tanks, airplanes, missiles and helicopters.

What is causing this outcome?

The Success of The German Wehrmacht

Serval years ago I read a book by two Swedish authors Michael Tamelander and Niklas Zetterling about the D-Day. One of the things they studied were the assumption that the Allied forces were superior to the Germans because the Allies won WW2. On closer examination, this was not true, and neither was it true during WW1. In both wars, the Axis powers were defeated primarily due to the superior production capacity, manpower, and resources accessible to allied powers. The book gives a surprising number of examples of how of German forces would win battles despite surprisingly large odds against them.

To understand the superiority of the German forces on the battlefield, the authors looked at how military training was done in Germany and how it was done in America and Britain. Allied soldiers were given problems to solve in the classroom. These were usually cases that involved tactics and strategy. The soldiers were given an hour to figure out the optimum solution to a challenge on the battlefield. If after the hour had passed, they gave a suboptimal answer they were asked to go back and think about the problem for a longer period of time.

I think this is a good analogy to the MIT software engineering philosophy of doing "the right thing" as described Richard Gabriel. The MIT approach emphasize correctness above all else. We could rephrase this type of thinking on the battlefield as an extreme emphasis on getting the correct or optimal solution in each battlefield scenario.

When German soldiers were presented with the same kind of battlefield challenges in the classroom, they were given a mere seven minutes to solve the same problem. Seven minutes! That is orders of magnitude shorter time. How could their teachers expect them to solve these problems in such a short time?

Simple! They followed the Unix philosophy of "worse is better". Their solution didn't have to be any good. It just had to be adequate. The one thing that trumped all other concerns was arriving at a solution in less than seven minutes. Spending more time than seven minutes was a failure. In other words, Germans were optimizing for quick decision-making while allies aimed for optimal decisions.

The results of this difference in philosophy had a catastrophic outcome for Allied forces. By the time Allied forces had pondered the situation on the ground properly, the realities had changed too much for their solution to be applicable. Germans soldiers, on the other hand, were very dynamic and always adapting to the current situation.

What Germans were doing is at the heart of iterative design. You make a choice which may not be optimal, but then always use feedback about the outcome of that choice to quickly change and adapt your tactics. That is a profoundly different approach to traditional large upfront design, where everything is analyzed in detail to reach an optimal solution.

With an iterative approach, we only care to be less wrong on each iteration. Every iteration should be an improvement on the previous one. A battle is a series of iterations driven by feedback. Each time you make a decision on the battlefield, it will have consequences. After observing those consequences, you make the next decision.

Russian forces fighting in Ukraine today are facing much the same problem. Central command is moving soldiers into planned positions. It seems apt to quote Mike Tyson here:

Everyone has a plan until they get punched in the mouth

What Tyson observed in the ring holds true in battle: The most elaborate plans can quickly fall apart as you face the enemy punching you in the face. In such a situation, it is not the quality of your carefully laid plans which matters, but your ability to react to challenges as they appear.

Let us jump to an entirely different topic to see the same forces at play.

Evolution vs. Intelligent Design in Biology

I believe evolution presents the ideas around iteration and feedback in their purest form. It is counterintuitive that a process such as evolution is capable of producing systems which are far more complex and advance than anything intelligent design has been able to do. By intelligent design, I mean an intelligent human being or teams of humans creating a system or machine. Evolution is not intelligent. Evolution cannot look ahead and see potential problems with a particular design. Essentially, evolution churns out numerous half-baked ideas, which are incremental modifications of existing solutions. Then evolution checks what modification sticks.

The reason this works so well is because of the feedback system. People who don't understand evolution very frequently overlooked this crucial part. If evolution was just random changes to DNA, one would never have been able to produce any organism.

The same ideas pop up in various software development philosophies. Let us explore the "worse is better" ideas further.

The Cathedral and the Bazaar Open-Source Development

Open-source guru Eric S Raymond coined the term cathedral and the bazaar. Like evolution, he showed that by using a lot of mediocre software developers, coupled with frequent releases, plenty of users and lots of feedback, higher-quality software could be built than by a small team of exceptional developers. This team of intelligent and extremely capable developers represents intelligent design in the analogy we are building.

When you designed the system, no matter how well you have thought about it you are likely to get some things wrong. If you spend a long-time thinking and planning the solution, there is a high chance that you veered off in the wrong direction. Evolution and the bazaar mode of development might not create very good solutions at each iteration. However, each iteration is very short and feedback is given quickly, so development can be adjusted before any problems compound themselves.

Humans have all kinds of design flaws, which exist because evolution cannot look far ahead. Evolution is always making small adjustments to what it currently has. That is why we have ended up with stupid designs like the cables going into the sensors of our eyes being connected at the front so that we get a blind spot. It is also why our spine is poorly suited for walking on two legs. However, there's little evolution can do about that because we started as four-legged creatures and were adapted from that initial starting-point.

We could compare this process with software development by considering developing a computer game, and we have the code for a four legged animal roaming around in our game world. Then we are given the task to add two legged humanoids to the game. There are two ways we could go about this. We could create the code for the two legged creature from scratch with some careful planning, or just start hacking the code for the four legged creature until we get what we want. Hacking requires less sophistication, while creating the code from scratch requires developers with more skill.

We could actually see how something very much like this played out in real life with the Tanenbaum–Torvalds debate about Microkernels vs. Monolithic Kernels. Or just look at HURD vs. Linux. The ragtag group of hackers of varying quality managed to create a better product than elites designing the HURD microkernel.

Market Economy vs. Planned Economy

No person or government bureau plans how many toilet paper rolls, bread, or cars should be produced in market economies. Figuring out how much should be produced of each item in a modern economy is an extremely complicated problem. And yet, the problem is being solved by a surprisingly simple process.

1. If more people start demanding a product, then increase the price.

2. If fewer people start demanding it, then decrease the price.

The first rule spurs companies to try to make more of the product because they can make more money. The second induce them to cut production. Like evolution, a set of very simply rules are used to produce a very complex system.

Today, we are conditioned to think that a market economy is superior to a planned economy. It is hard to appreciate how much sense planned economy makes intuitively. With a planned economy, one can think about the economy and what should be produced for years ahead. For a while, this actually works. Countries with planned economies were able to industrialize and grow their industrial production of an astonishing rate for several years. The reason, a planned economy, was able to channel much higher levels of resources into building factories and avoid duplication of effort. In a market economy, countless companies are spending resources to make almost the same product. At the face of it, that seems very wasteful.

However, planned economies ultimately fails to a large degree because of the information problem. Economists Joseph E. Stieglitz and Paul Krugman among others have some interesting coverage of this topic. As economies get more advanced, and there are more products to decide production levels for, the number of variables in your equations explode. Solving the equation or even getting the information needed to input into the equations become an impossible task. The result is the wrong level of production for all kinds of things. The Soviet Union was constantly plagued by overproduction of certain items and severe shortage of other random products.

This problem is perhaps not that different from the one faced by the software architects trying to gather all kinds of information about requirements of the software system that they will design. Like the Soviet planners, they will ultimately fail in getting all the information correct. Market economies and get stuff wrong all the time as well, but they get feedback quickly based on real usage to allow rapid adjustment.

Although I should point out that there are many cases where a market-based approach doesn't work because the feedback system itself is broken.

Toyota Production System vs. Ford

There are plenty of differences between the Toyota and Ford production systems, but I will concentrate on those parts relevant to this particular discussion. The Ford approach to mass production is similar to a planned economy in that somebody at the top tries to figure out all the details of the operation. There are few local decisions, as in a free market economy. Like intelligent design, one makes a large upfront design where one optimizes the utilization of every machine.

This is done by for instance placing a large buffer of parts at each machine in case there is a failure at one of the preceding machines at the assembly line. Thus, the assembly line can keep going even when other machines fail.

The Toyota production system is more similar to evolution and the market economy. Failure is celebrated because it gives valuable feedback which will help us improve. Batch sizes between machines are made small so that if any machine fails, big chunks of the assembly line are brought to a halt. That choice brings focus to problems in the production system. This strategy is coupled with encouragement towards regular workers to help make things get back on track. This approach is what we call continuous improvement. So, essentially, you get a manufacturing system which is not set in stone but which improves through multiple iterations directed by ample feedback.

This approach is analog to the Cathedral vs. the Bazaar in that mediocre or average developers are relied upon actively, not just the elite programmers. Workers are helping to make the assembly line work better. It is not just about the specialists and the engineers at the top, as would be the case with the Ford production system.

The Toyota production system is part of a larger trend that existed among Japanese Automakers referred to as lean or agile. These ideas would later inspire several different ideas around how to organize software development.

A Falcon 9 rocket in hangar. Official SpaceX picture cropped.

SpaceX Approach To Rocket Design

NASA is building the Space Launch System (SLS) in a very incremental manner. They make one piece at a time and test it thoroughly. Everything is planned in the smallest detail.

SpaceX in contrast is forging ahead quickly using an iterative approach. Instead of building complete parts, they make numerous prototypes of their Starship rocket, which they launch. Next, they observe what happens at launch. It is what we call a hardware-rich approach.

Yet, we can see many relations to other approaches discussed earlier. NASA is following the meticulous MIT style approach where making every part correct is paramount.

Elon Musk in contrast is forging ahead with a "worse is better" approach. It doesn't matter that the first Starship versions are flawed in some way. He builds them quickly that he can launch them and gain feedback to power the whole development process.

Let me give an example of this process from the perspective of a former NASA employee, Dan Rasky who joined SpaceX. One of his first interactions with Elon Musk was when determining how to make their PICA (Phenolic Impregnated Carbon Ablator) heat shield for the Dragon capsule.

Read more: SpaceX Leaves Searing Impression on NASA Heat Shield Guy.

The issue they were considering was whether to buy PICA or manufacture it in-house. Such a decision would have taken months of back and forth, according to Dan Rasky. Every possible problem and benefit would be examined in detail. Elon Musk instead basically had one meeting and when Dan Rasky recommended manufacturing in-house, then Elon Musk decided to follow his recommendation on the spot and concluded the meeting. Dan Rasky remarked he had never experienced anything like that. His expectation was a couple of months of further discussion and analysis to be absolutely certain in-house manufacturing of PICA was the right choice.

The SpaceX approach was to build smaller prototypes of the PICA factory rather than doing a lengthy theoretical analysis. Lessons from each prototype would be fed into the next larger model until they had a full scale working factory.

A problem with a length upfront design is that once you begin building the full solution and discover a problem, it may get too costly to change direction.

The Marshmallow Challenge

Peter Skillman came up with a design exercise called the Marshmallow Challenge. The goal was simple:

In eighteen minutes, teams of four must build the tallest freestanding structure out of 20 sticks of spaghetti, one yard of tape, one yard of string, and one marshmallow. The marshmallow has to be on top.

It is harder than it sounds. Among the interesting results from this was:

Recent MBA graduates did worst at this exercise. Among the best performers were kindergarden graduates.

The reason for this outcome was that the children used the principle of iterative development, while the MBA students focused on developing and executing the one true solution. Hence, when they finally put the marshmallow on the top; the structure would collapse because they had not thoroughly understood how to create a stable structure. The kids, however, started putting on the marshmallow very early on and learned from mistakes.

In other words, the MBA students were using “intelligent design” or acting like Allied soldiers in WW2, while the kids were using an evolutionary strategy and acting more like the German Wehrmacht or the Toyota Motor Corporation.

Scrum vs. Waterfall in Software Development

Scrum is one of many agile methods of software development inspired by lean manufacturing among Japanese automakers. These methods are often contrasted with what is called Waterfall method of development. The Waterfall approach means large upfront design and planning. A bit like how allied forces in WW2 would plan an attack or how Russian generals today plan attacks in Ukraine, rather than adapting to the situation on the battlefield as they evolve.