Yes it’s that old knotty problem again! Over at gapingvoid.com Hugh MacLeod is fond of using Venn diagrams to illustrate overlapping concerns so here’s one that I recently used for addressing the eternal architecture versus design debate that was the source of much discussion at a recent Architecture Thinking class I was giving.

As I remember it the discussion went something like this:

• Student: So what’s the difference between architecture and design? It seems from what you’re saying its just a matter of scale?
• Me: Whilst its true to say that architecture addresses the major components of the system, rather than the detail, it’s more than that. The architecture is the bridge between the “what” (that is the requirements) and the “how” (that is the design).
• Student: Yeah but isn’t that we usually call “high-level design”.
• Me: Not really. Grady Booch says: “All architecture is design but not all design is architecture”. (I cheated and looked up this quote afterwards and found that Booch goes on to say “architecture represents the significant design decisions that shape a system, where significant is measured by cost of change.”). In my experience high-level design is just the the view that allows the complete system to be represented on one page.
• Student: I still don’t see what the difference really is.
• Me: Okay, here’s the real difference for me (at this point I draw the above Venn on a flip-chart). As well as defining the structure of the system, architecture must also embrace the “what” and the “how” of that structure.  The “what” in this context is the requirements (functional and non-functional) and so architecture involves reasoning about and resolving these sometimes conflicting requirements. It’s about addressing those architecturally significant requirements (the “what”) that will drive (and constrain) the “how” (the design).

Now, if I was drawing this again (and maybe I’ll do this next time) I would actually draw a third overlapping circle which I’d label the “why”. This is where we’d capture the rationale for why we make the (architectural) decisions we do.

Thanks Hugh, this is a neat way of explaining the way things are!

Clay Shirky has written a great entry on his blog called The Collapse of Complex Business Models which has set me thinking about the whole issue around complexity; especially as it applies to complex software systems.The article uses Joseph Tainter’s book called The Collapse of Complex Societies for the basis of its premise. In that book Tainter looked at various ancient, sophisticated societies that suddenly collapsed (the Romans and the Maya for example). Tainter postulated that these societies “hadn’t collapsed despite their cultural sophistication, they’d collapsed because of it”. His theory was that as societies become more organised and efficient they find themselves with a surplus of resources and managing this surplus makes the society more complex. The spare resources go more into “gilding the lily” than creating what is strictly required. Early on the value of this complexity is positive and often pays for itself in improved output. Over time however the law of diminishing returns reduces this value and eventually disappears completely at which point any additional complexity is pure cost. The society has then reached a tipping point and when some unexpected stress occurs it has become too inflexible to respond. As Tainter says when “the society fails to respond to reduced circumstances through orderly downsizing, it isn’t because they don’t want to, it’s because they can’t”.

Shirky’s theory is that today, the internet means many businesses are facing similar challenges: adapt to a new way of working or die. In particular the media industry is failing to recognise it has built hugely complicated edifices around the production of media (that’s TV, movies, newspapers and music) that the internet is wiping away. Media execs like Rupert Murdoch are looking for ways of maintaining their status quo by just using the internet as a new delivery channel which allows them to continue with their current, costly and complex, business models. What they don’t realise is that the internet has changed fundamentally the way the media industry works with practically zero production and delivery costs and unless they change their complex and expensive ways the old order will die out.

So what’s this got to do with software systems? Although we might think the systems we are building today are complex we are about to start building a level of complexity that is an order of magnitude (at least) above what it is today. If we are to address some of the worlds really wicked problems then we need to make systems that are not just the siloed systems we have today but that are systems-of-systems interconnected in ways we cannot yet imagine or envisage. Whilst such interconnected systems might enable collaborations that help solve problems we should remain aware that we are adding new levels of complexity that may be hard to manage and even harder to do without if we ever hit some unexpected “stress” situation. In 1964, science fiction author Arthur C. Clarke wrote a short story called “Dial F for Frankenstein”. In the story the phone network (this was before the internet had even been thought of although, interestingly Tim Berners-Lee, the inventor of the web, suggested this was the story that anticipated that technology) had become so large and complex it was effectively a giant brain that becomes self-aware. Not only could it not be turned off, it started to think and eventually took over the world! Clarke himself even says “Dial F for Frankenstein is dated now because you no longer dial of course, and if I did it now it wouldn’t be the world’s telephone system it would be the internet. And that of course is a real possibility. When will the internet suddenly take over?”

We should be very aware therefore, if we are to learn anything from the history of those ancient civilisations, that adding more and more complexity to our systems is not without cost or risk. Although in the short-term we may reap rewards, in the longer terms we may yet regret some of the actions we are about to take and therefore make sure we remember to provide an on/off switch for these systems!

Whilst teaching an IT architecture class in Moscow this week I was asked the above question by one of the students. The question was in relation to projects and when you should realise the project is not going to deliver and should therefore be cancelled rather than carrying on regardless. I hate using the phrase “it depends” in answer to a question like this but this was one case when that’s all I could think of! However, having had a bit of time to reflect on this (on a long journey back home) here are a few thoughts that I’ve had since then. This is also related to my previous blog entry.

In some ways working on a software delivery project is a bit like fighting a war (though admittedly you don’t usually risk losing your life). In a war you may not win every battle and may sometimes need to retreat and reconfigure however, the important thing is to focus on the strategic battles that will lead you to get to the overall objective (and win the war). In a software delivery project winning the war is to successfully deliver the system or application on time and within budget. The battles are what the architect and project manager must fight every day to address the technical and other issues which arise and must be overcome for fear of letting the project get out of control. So what are the battles, which if not won, are going to cause the project to fail and, more importantly for this topic, will lead to surrender if the cost gets too high? Here’s my top five in order of importance:

1. No, or poor, sponsorship.If there is no clear (and strong) sponsorship from the right level in the enterprise then you may as well pack up and go home immediately. Even if the project does get to the delivery stage the chances are no one will really want or need the system. The job of the sponsor is to evangelise, convince and cajole a sometimes reluctant workforce of the need for the new system and the change that it might bring about. Weak sponsors will be unable or unwilling to do this.
2. Not having a water tight business case.If the business case for why the new system is needed is not well thought through with a clear cost-benefit analysis then there is a danger that the project could be cancelled at any time. I once worked on a project that was cancelled days before we were due to exit system test because a review of the business case revealed a flaw in the return on investment (ROI) of the new system so the client decided to pull the plug on the project rather than deliver something that would not make the cost savings that were expected of it. Painful but probably the right decision!
3. Not engaging all stakeholders. Not identifying and engaging (i.e. talking to, convincing, schmoosing, wining and dining or whatever it takes) all the stakeholders is a bit like death by a thousand cuts. Although you will start off well, entropy will gradually set in and and things will start to fall apart. Stakeholders who should have been informed early on will learn about the project through the grapevine and will not only not support it but may actively try to kill it. They may see it as a threat or may treat it as sour grapes, “no one bothered to tell me about it so why should I support this”?
4. Underestimating the complexity of building interfaces to legacy systems. Very few projects have the luxury of a clean sheet of paper when starting out. There is nearly always some old, creaking legacy system that needs to be interfaced with, even short-term or temporarily. In my experience one should never underestimate the complexity of these legacy system interfaces. And I don’t just mean technology interfaces. Many times IT departments will see it as a threat to their existence that what starts out as an interface may ultimately lead to a complete replacement of the system that they run and manage so lovingly and will therefore do whatever they can to derail the project.
5. Not having regular and viable releases of the system/application. Gone are the days when a project could go off for many years before delivering anything of value to the key stakeholders (or I hope they have). I cannot emphasise enough the importance of delivering something of value that users can get their hands on and starting using that will give them some business benefits. This not only shows them the value of the new application but also gives them a belief that IT can actually deliver stuff and will also get them to “buy-in” to the new system and may well have the useful side-effect that they become sales-people for the new system and will convince colleagues of its benefits.

I believe that if one or more of these issues exist on your project you should either take very serious steps to address them or consider looking for another job because there is a greater than 50% chance everything is going to get very messy and is going to end in tears or much, much worse.