Interprise Architecture and Ultra-Large-Scale Systems

In a previous post I introduced the term “Interprise Architecture” to describe how the internet is breaking down the traditional boundaries of the enterprise and thus requires a new approach to Enterprise Architecture that’s not just about describing what’s inside the enterprise but also what’s on the outside. No longer can Enterprise Architects create blueprints for some future state that the enterprise will one day reach with roadmaps for how that state will be achieved. There are too many disruptive influences and new technologies that are impinging on the enterprise that will not only mean the roadmap is sending you in the wrong direction but that you are probably using the wrong mode of transport to get there as well.

I received a few comments on this from folk at the Software Engineering Institute (SEI)as well as Gartner. The work on Ultra-Large-Scale (ULS) Systems from the SEI particularly drew my attention and resonates nicely with some of my own thoughts. Here are some of the key ideas from the SEI report Ultra-Large-Scale Systems – The Software Challenge of the Future plus some additional musings of my own on what constitutes Interprise Architecture. First, ULS:

  • The SEI report on ULS systems was funded by the United States Department of Defence (DoD) which asked the SEI to consider future systems that could not only contain of billions of lines of code but also exhibit some, possibly all, of the following characteristics: decentralisation; conflicting, unknowable, and diverse requirements; continuous evolution and deployment; heterogeneous and changing elements; erosion of the people/system boundary; and normal failures of parts of the system.
  • ULS systems are likely to mean that traditional software and systems engineering approaches will no longer be adequate or can be the primary means by which such systems are designed (architected) or built.
  • ULS systems can be compared with cities whereas traditional systems can be compared with buildings. Buildings can be designed and built to a blueprint whereas cities emerge and are continuously adapting over time.
  • ULS systems are comprised of a dynamic community of interdependent and competing organisms (in this case, people, computing devices, and organizations) in a complex and changing environment. These are referred to as socio-technical ecosystems.
  • ULS systems are ones that are continuously evolving with new behaviour constantly emerging. In this respect they have the attributes of wicked problems where the problem is never definitively solved (or indeed understood).
  • ULS systems expect failure to be the norm and that unusual situations and boundary conditions will occur often enough that something will always be failing.

The SEI report is primarily aimed at allowing the US military to develop new systems however I believe the key ideas that challenge the development of such systems also have wide applicability in business systems, the sort I’m most interested in. I see that what I have characterised as Interprise Architecture could therefore be applied to developing ULS business systems. Here are three examples of ULS business systems that might benefit from an Interprise Architecture approach:

  • Electronic Trading Systems. These are systems that trade securities (such as stocks, and bonds), foreign currency, and exchange traded derivatives electronically. They use IT to bring together buyers and sellers through electronic media and create a virtual market place. Such systems are typically built using proprietary software that has grown and evolved over many years. Investment banks have extremely complex technology requirements, as they have to interface with multiple exchanges, brokers and multi-dealer platforms, as well as their own pricing, profit and loss (P&L), trade processing and position-keeping systems. The challenge here then is not only the large numbers of systems but also the increasing complicated regulatory environment.
  • Electricity Generation and Metering. The generation and consumption of electricity faces a number of unique challenges including improved and more efficient use of green technologies as well as smart metering. Traditional electrical meters only measure total consumption and as such, provide no information of when the energy was consumed. Smart meters provide an economical way of measuring this information, allowing price setting agencies to introduce different prices for consumption based on the time of day and the season.
  • Retail Systems. As retailers look for ever more cunning ways to get consumers to part with their hard-earned cash, traditional (i.e. high street) and electronic retail will merge more and more. For example not only can I use my 3G enabled smart-phone from the store I happen to be in to quickly compare prices in other stores in the area, the store itself can potentially detect I am shopping there using location based services and make me an enticing offer to shop there.

So here are the seven challenges that I see Interprise Architecture must deal with in developing a ULS business system:

  1. Requirements are unknowable. Sometimes the very act of capturing a requirement (in whatever form) changes the nature of that requirement. Interprise Architecture must not only allow for continuously changing requirements but must also recognise that some uses of the system cannot be known up-front; hence the need to build more adaptable systems.
  2. The boundary between people and systems is at best blurred and at worst never established. Sometimes people will be users of the system, sometimes they (or at least the devices they own) will be part of the system.
  3. Development never stops but is a continuous cycle. Development processes as well as the projects that deliver such systems must therefore support this never-ending cycle.
  4. Systems continuously adapt and exhibit emergent behaviour. As new uses of the system are “discovered’ by users the components that make up the system need to be able to adapt to satisfy those new behaviours.
  5. Failure (of parts of the system) is inevitable. Just like a fire in a building in a city can be localised and extinguished without by and large affecting the whole of a city then so to must Interprise Architecture allow for partial failure and reconfiguration of some components.
  6. Development tools and languages need to take account of the unpredictable and maybe even unspecifiable aspects of systems development. Traditional development tools favour left-brain thinkers where logic and reasoning can be applied to develop systems that move from abstract ideas to physical implementations (from models to code if you like). New tools for developing and describing Interprise Architectures need to be able to inject a bit of right-brain thinking by allowing creative multi-disciplinary elements to be added.
  7. Governance needs to be de-centralised. Strong, top-down governance (the sort favoured by Enterprise Architects) cannot work in a system where all the parts may not even be known. Interprise Architecture needs to recognise that some components are outside its control or immediate sphere of influence and have policies in place that allow new components to be added which don’t harm or damage the whole system.

As an interesting post-script to this the Financial Times recently published an article on Facebook and the plans that CEO Mark Zuckerberg has for advancing his brainchild. Zuckerberg had just announced a new feature on Facebook called Deals which allows smartphone users who have downloaded the Facebook application to check in at a physical location such as a coffee shop and get a reward. Zuckerberg says:

If you look five years out, every industry is going to be rethought in a social way. You can remake whole industries. That’s the big thing.

Facebook is one example of how external applications that allow users to impinge on the enterprise are changing how Enterprise Architects must think.

Next, a story for what a ULS business system might look like and how it might work.

Enterprise Architecture is Dead. Long Live Interprise Architecture

ZapThink have recently been drip-feeding a series of ZapFlashes predicting the end of Enterprise Architecture (possibly allowing them to derive some further consulting/education business by creating a little fear, uncertainty and doubt). Here are some of their predictions and my thoughts. Comments welcome.

  • The secret to 21st century software innovation. Is, according to ZapThink, in harnessing the power of complex systems (i.e. rather than just enterprise-wide systems). A complex system is one that shows some form of emergent behaviour where the properties of the system as a whole exhibit behaviour that the individual components do not. Most people don’t get this confusing systems that are just complicated with ones that are complex. For me this is spot on. As the enterprise morphs into basically the whole of the internet traditional governance models breakdown. The boundaries of the enterprise can no longer be nicely defined and who is in the enterprise and who is not becomes harder to pin down. People who get this and who can successfully harness the “interprise” (my new word of the month) by using the positive effects of social networking etc will win out over the next few years. Of course this involves huge risks not least of which around how will enterprises keep control of their data and intellectual capital.
  • RIP enterprise software. Enterprise software as provided by the large package providers is seen as large and inflexible and not delivering on the benefits promised by SOA. Companies are finding themselves encumbered with expensive and hard to maintain software they can’t “bend” to do what they want. ZapThink’s take is that whilst enterprise software may have failed SOA has also failed to deliver on its promise of providing flexible business processes that can be quickly adopted to new needs. My take is that like everything else in our industry nothing is given chance to bed in before the next wave comes and sweeps everyone along with it. Many clients I see are still grappling with getting an effective process in place for developing traditional systems let alone service-based ones and definitely cannot deal with complex systems properly.
  • The beginning of the end for Enterprise Architecture frameworks. Architecture frameworks (especially enterprise ones like Zachman and DoDAF) are counterproductive to developing an effective EA strategy. These frameworks are inflexible, sometimes encouraging what ZapThink refer to as checklist architecture. Checklist architecture focuses on achieving goals laid down by the architecture framework rather than the changing needs of the business. True but having no framework at all leads to even greater chaos in my experience. A framework is a structure which is meant to contain relevant guidance and work products to deliver real-world solutions. Frameworks that become too theoretical are always doomed to fail as they should. I hope this forms the basis of something more relevant to the real world.

So how would I characterise “interprise architecture”?

  1. Interprise architecture recognises that it is not enterprise architecture that is dead but trying to constrain architecture by the bounds of the enterprise that is no longer achievable. The ‘architecture’ bit still applies, but not the ‘enterprise’ bit.
  2. Governance models need to recognise that this extended enterprise includes people and other systems that are not controlled by the IT department (people using social networking software who will periodically ‘overlap’ with the enterprises systems).
  3. The ‘architecture’ bit needs to take into account the fact that systems are complex (in the emergent sense of the word) and it’s not always possible to tie down the requirements in a nice orderly way. Indeed fully defining the requirements may be counter-productive and not allow emergent behaviour. Processes for developing such complex systems need to take this into account.

To be sure this is not a complete list. This idea is not yet fully-formed in my own mind and needs a bit more time to ’emerge’ properly. Watch this space as they say.