Question: How many engineers does it take to change a light bulb?

Answer: Some joke which is funnier the more people that are needed for the job and the more convoluted the reason for their required presence.

Did you ever stop to think why there are so many gags on this theme?

I think it’s because we can all somehow appreciate the inherent comedy in how engineers systematically over-provision, create fallbacks, fail-safes and redundancies at every possible corner.

I’ve been thinking about that over the last couple of weeks, since the neighbourhood I live in has been interrupted by massive construction work.

In the first wave, men wearing hi-viz vests manoeuvring gigantic machines ripped the top asphalt layer off of main street and replaced it with fresh new blacktop.

Then a few weeks later, a different crew turned up and started ripping away the brand new asphalt in broad stripes on both sides of the road, only to fill the resulting cavities with some kind of material that behaved like asphalt and smelled liked asphalt only it was red.

The new colour asphalt was meant to indicate bike lanes, which, being an avid biker, I’m all for.

I never understood however, why they couldn’t simply have painted lanes on top of the brand new road, instead of letting tons of perfectly good asphalt go to spoil. I also didn’t grok why two different sets of yellow vested men had to close the main thoroughfare for traffic twice, instead of just getting the job done in one fell swoop.

But then I’m not a schooled engineer.

If I were, I’m sure the operation would have made perfect sense. Where other people see blatant wastefulness, I assume I would have seen redundancy. I would have probably leant on the theory of WIP limiting, which says that in the long run any project will benefit from putting a cap on the number of processes that can run in parallell.

That is to say: If you think of all the activities that goes into a project as occupying separate rows, there should be a limit to the number of such rows that are allowed to overlap; a limit to Work In Progress.

Advocates of this concept are convinced that it’s worth whatever tradeoffs you have to make. If a frugal WIP limit means parts of a development team has to idle during a sprint because there’s no work for them to do, then that’s considered a bonus, since the slack can be used to read up on new technologies. The idea is that over time this will lead to boosted productivity.

If imposing limits on the number of processes that can be in progress simultaneously is meant to generate redundancy, then what is the overarching purpose of redundancy?

In a word: resilience.

In her posthumously published gem of a book Thinking in Systems, the scientist and writer Dana Meadows notes resilience as one of three characteristics of highly functional systems. The other two are self-organisation, and hierarchy.

Resilience, as defined by the dictionary, is “The ability to bounce or spring back into shape, position etc., after being pressed or stretched. Elasticity. The ability to recover strength, spirit, good humor, or any other aspect quickly.”

Meadows describes how resilience is generated by “a rich structure of feedback loops, operating through different mechanisms, at different time scales, and with redundancy – one kicking in if another fails – all working in different ways to restore a system even after a large perturbation.”

An important contribution that the field of system dynamics made, was to distinguish between stability and resilience. Just as a living system is only stable once it’s dead, an organisation – or a project – will become fragile if it’s put in a straightjacket of constancy.

In contrast, a resilient system “can be very dynamic. Short-term oscillations, or periodic outbreaks, or long cycles of succession, climax and collapse may in fact be the normal condition, which resilience acts to restore!”

One manifestation of the difference between stability and resilience, is the fact that the former is easily observable while the latter often remains invisible until you exceed its limits. A very concrete example of observable stability, is the velocity metric that any self-respecting scrum team is keeping track of.

Velocity is a nominal value for the productivity of a development team. Nominal because it’s measured in so called “story points”, which is the teams local currency for assessing complexity levels of different engineering tasks. Fixing a cosmetic bug in the user interface: 0.5 points; migrating to a new framework: 35 points, and so on so forth.

The team’s process leader – the scrum master – keeps track of the number of points the team commit to in any given sprint, as well as how many they end up delivering. According to common agile wisdom, a good team is one that produces a stable average. They may not be the fastest but they’re dependable, you know with mathematical certainty when they’ll get a project delivered. That is an example of stability quantified.

Resilience, par contre, is a very different animal.

Meadows had a beautiful way of describing it:

“I think of resilience as a plateau upon which the system can play, performing its normal functions in safety. A resilient system has a big plateau, a lot of space over which it can wander, with gentle, elastic walls that will bounce it back, if it comes near a dangerous edge. As a system loses its resiliences, its plateau shrinks, and its protective walls become lower and more rigid, until the system is operating on a knife-edge, likely to fall off in one direction or another whenever it makes a move. Loss of resilience can come as a surprise, because the system usually is paying much more attention to its play than to its playing space. One day it does something it has done a hundred times before and crashes.”

A ‘system operating on a knife-edge‘. That’s a poetic way of describing how the agile philosophy tend to see development teams.

Randomness is the name of the game. Anything can happen. The roadmap can be redrawn from scratch at the next planning session since the suits upstairs decided to “pivot”. Or the deadline that was three month out can move to next week. Life in the trenches is unpredictable.

Which means it makes a lot of sense to want to build in buffers, to strive for stability, even at the price of risking local optimisation.

What Meadows has to say about this seemingly unsolvable problem is that it comes down to how we think of boundaries.

The important thing to keep in mind with boundaries, is that they’re always artificial. Which means they can be redrawn by humans.

Within the strict boundaries of my neighbourhood, perhaps it made sense to over-provision in order to get perfect bike lanes, but what about the poorer community next-door where the roads remain full of pot holes? That area was probably out of bounds, for whatever reason. Another tax district, someone else’s problem, what do I know.

Redrawing boundaries will invariably paint new pictures of what problems are relevant to solve.

It also puts into question how well it makes sense to solve them.

If you want to colonise Mars, every single piece of the puzzle must indeed be flawless. In other cases however, like if you want to bridge the gaps between neighbouring communities, perhaps good enough is perfect.

While rethinking and transcending boundaries might be the ideal however, in reality it’s also a stumbling block, not least on a very human level.

We need cognitive and conceptual boundaries in order to feel comfortable. In Meadow’s words: “There’s something within the human mind that is attracted to straight lines and not curves, to whole numbers and not fractions, to uniformity and not diversity, and to certainties and not mystery.”

But. She also goes on to say that there is “something else withins us that has the opposite set of tendencies, since we ourselves evolved out of and are shaped by and structured as complex feedback systems. Only a part of us, a part that has emerged recently, designs buildings as boxes with uncompromising straight lines and flat surfaces. Another part of us recognises instinctively that nature designs in fractals, with intriguing detail on every scale from the microscopic to the macroscopic. That part of us makes Gothic cathedrals and Persian carpets, symphonies and novels, Mardi Gras costumes and artificial intelligence programs, all with embellishments almost as complex as the ones we find in the world around us.”

So what can we hope for, at the end of the day? Will we ever be able to beat rigidity into submission and come out the other side with true mastery of a system as boundless as the universe itself?

Meadows didn’t think so, but she did have a good alternative suggestion:

“We can’t control systems or figure them out. But we can dance with them! I already knew that, in a way. I had learned about dancing with great powers from whitewater kayaking, from gardening, from playing music, from skiing. All those endeavours require one to stay wide awake, pay close attention, participate flat out, and respond to feedback. It had never occurred to me that those same requirements might apply to intellectual work, to management, to government, to getting along with people.”

PS

As with any proper system, this blog is starting to get its fair share of interconnections. Here follows a few previous posts which touches on the topics mentioned in this text.