Clay Shirky has written a great entry on his blog called The Collapse of Complex Business Models which has set me thinking about the whole issue around complexity; especially as it applies to complex software systems.The article uses Joseph Tainter’s book called The Collapse of Complex Societies for the basis of its premise. In that book Tainter looked at various ancient, sophisticated societies that suddenly collapsed (the Romans and the Maya for example). Tainter postulated that these societies “hadn’t collapsed despite their cultural sophistication, they’d collapsed because of it”. His theory was that as societies become more organised and efficient they find themselves with a surplus of resources and managing this surplus makes the society more complex. The spare resources go more into “gilding the lily” than creating what is strictly required. Early on the value of this complexity is positive and often pays for itself in improved output. Over time however the law of diminishing returns reduces this value and eventually disappears completely at which point any additional complexity is pure cost. The society has then reached a tipping point and when some unexpected stress occurs it has become too inflexible to respond. As Tainter says when “the society fails to respond to reduced circumstances through orderly downsizing, it isn’t because they don’t want to, it’s because they can’t”.

Shirky’s theory is that today, the internet means many businesses are facing similar challenges: adapt to a new way of working or die. In particular the media industry is failing to recognise it has built hugely complicated edifices around the production of media (that’s TV, movies, newspapers and music) that the internet is wiping away. Media execs like Rupert Murdoch are looking for ways of maintaining their status quo by just using the internet as a new delivery channel which allows them to continue with their current, costly and complex, business models. What they don’t realise is that the internet has changed fundamentally the way the media industry works with practically zero production and delivery costs and unless they change their complex and expensive ways the old order will die out.

So what’s this got to do with software systems? Although we might think the systems we are building today are complex we are about to start building a level of complexity that is an order of magnitude (at least) above what it is today. If we are to address some of the worlds really wicked problems then we need to make systems that are not just the siloed systems we have today but that are systems-of-systems interconnected in ways we cannot yet imagine or envisage. Whilst such interconnected systems might enable collaborations that help solve problems we should remain aware that we are adding new levels of complexity that may be hard to manage and even harder to do without if we ever hit some unexpected “stress” situation. In 1964, science fiction author Arthur C. Clarke wrote a short story called “Dial F for Frankenstein”. In the story the phone network (this was before the internet had even been thought of although, interestingly Tim Berners-Lee, the inventor of the web, suggested this was the story that anticipated that technology) had become so large and complex it was effectively a giant brain that becomes self-aware. Not only could it not be turned off, it started to think and eventually took over the world! Clarke himself even says “Dial F for Frankenstein is dated now because you no longer dial of course, and if I did it now it wouldn’t be the world’s telephone system it would be the internet. And that of course is a real possibility. When will the internet suddenly take over?”

We should be very aware therefore, if we are to learn anything from the history of those ancient civilisations, that adding more and more complexity to our systems is not without cost or risk. Although in the short-term we may reap rewards, in the longer terms we may yet regret some of the actions we are about to take and therefore make sure we remember to provide an on/off switch for these systems!