sdlc

Software Developments Best Kept Secret

Peter Cripps comment , , ,

A few people have asked what I meant in my previous entry when a said we should be “killing off the endless debates of agile versus waterfall.” Don’t get me wrong, I’m a big fan of doing development in as efficient a way as possible, after all why would you want to be doing things in a ‘non-agile’ way! However I think that the agile versus waterfall debate really does miss the point. If you have ever worked on anything but the most trivial of software development projects you will quickly realise that there is no such thing as a ‘one size fits all’ software delivery lifecycle (SDLC) process. Each project is different and each brings its own challenges in terms of the best way to specify, develop, deliver and run it. Which brings me to the topic of this entry, the snappily titled Software and Systems Process Engineering Metamodel or ‘SPEM’ (but not SSPEM).

SPEM is a standard owned by the Object Management Group (OMG), the body that also owns the Unified Modeling Language (UML), the Systems Modeling Language (SysML) and a number of other open standards. Essentially SPEM gives you the language (the metamodel) for defining software and system processes in a consistent and repeatable way. SPEM also allows vendors to build tools that automate the way processes are defined and delivered. Just like vendors have built system and software modeling tools based around UML so too can vendors build delivery process modeling tools built around SPEM.

So what exactly does SPEM define and why should you be interested in it? For me there are two reasons why you should look at adopting SPEM on your next project.

  1. SPEM separates out what you create (i.e. the content) from how you create it (i.e. the process) whilst at the same time providing instructions for how to do these two things (i.e. guidance).
  2. SPEM (or at least tools that implement SPEM) allows you to create a customised process by varying what you create and when you create it.

Here’s a diagram to explain the first of these.

SPEM Method Framework
The SPEM Method Framework represents a consistent and repeatable approach to accomplishing a set of objectives based on a collection of well-defined techniques and best practices. The framework consists of three parts:
  • Content: represents the primary reusable building blocks of the method that exist outside of any predefined lifecycle. These are the work products that are created as a result of roles performing tasks.
  • Process: assembles method content into a sequence or workflow (represented by a work breakdown structure) used to organise the project and develop a solution. Process includes the phases that make up an end-to-end SDLC, the activities that phases are broken down into as well as reusable chunks of process referred to as ‘capability patterns’.
  • Guidance: is the ‘glue’ which supports content development and process execution. It describes techniques and best-practice for developing content or ‘executing’ a process.

As well as giving us the ‘language’ for building our own processes SPEM also defines the rules for building those processes. For example phases consist of other phases or activities, activities group tasks, tasks take work products as input and output other work products and so on.

This is all well and good you might say but I don’t want to have to laboriously build a whole process every time I want to run a project. This is where the second advantage of using SPEM comes in. A number of vendors (IBM and Sparx to name two) have built tools that not only automate the process for building a process but which also contain one or more ‘ready-rolled’ processes to get you started. You can either use those ‘out of the box’, extend them by adding your own content or start from scratch (not recommended for novices). What’s more the Eclipse foundation have developed an open software tool, called the Eclipse Process Framework (EPF) that not only gives you a tool for building processes but also comes with a number of existing processes, including OpenUP (open version of the Rational Unified Process) as well as Scrum and DSDM.

If you download and install EPF together with the appropriate method libraries you can use these as the basis for creating your own processes. Here’s what EPF looks like when you open the OpenUP SDLC.

EPF and OpenUP

The above view shows the browsing perspective of EPF, however there is also an authoring perspective which allows you to not only reconfigure a process to suit your own project but also add and remove content (i.e. roles, tasks and work products). Once you have made your changes you can republish the new process (as HTML) and anyone with a browser can then view the process together with all of it work products and, most crucially, associated guidance (i.e. examples, templates, guidelines etc) that allow you to use the process in an effective way.

This is, I believe, the true power of using a tool like EPF (or IBM’s Rational Method Composer which comes preloaded with the Rational Unified Process). You can take an existing SDLC (one you have created or one you have obtained from elsewhere) and customise it to meet the needs of your project. The amount of agility and number of iterations etc that you want to run will depend on the intricacies of your project and not what some method guru tells you that you should be using!

By the way for an excellent introduction and overview of EPF see here and here. The Eclipse web site also contains a wealth of information on EPF. You can also download the complete SPEM 2 specification from the OMG web site here.

Architecture is Architecture?

Peter Cripps architecture ,

At OT 99 (that’s Object Technology 1999, now known as SPA for Software Process Advancement) I attended a session by Kent Beck called Software is Software – Beyond the Horseless Carriage. The basic premise of Kent’s talk was that it was about time the software business “grew up” and its practitioners recognise it for what it is, a discipline in its own right which no longer needs to continuously borrow terms and techniques from other industries and disciplines. The title of the session refers to the time when the automobile was first invented and people called them horseless carriages because horse-drawn carriages were the only frame of reference they had. Unfortunately, 12 years later, I don’t think we have quite got around to jettisoning our horseless carriages, especially in the upfront work that is done in trying to map out the major system components and their relationships, sometime referred to as architecture (a word which itself is borrowed from another profession of course). On the face of it this may not seem to be a problem; after all those other industries (civil engineering, auto-engineering even film making) have been around a lot longer and so must be able to offer good advice and guidance to the business of software mustn’t they? Actually, I think there is a problem and Kent Beck was, and still is, right.

  • The business of ‘making’ software is fundamentally different from any other human endeavor. Software is infinitely malleable and potentially changeable right up to (and sometimes after) it has gone into production. No other engineering discipline has that flexibility. At some point drawings and blueprints have to be signed off, factories and production lines have to be built, building sites prepared and production begun. After this any change becomes prohibitively expensive. With software the perception (and sometimes the reality) is that code changes can be made right up to the moment the software ships.
  • Most other engineering disciplines have fairly well defined job roles, often with their own professional organisations, training programmes and qualifications and well understood and mature tools. These roles are usually carried out by separate individuals (in the construction industry it’s unlikely the architect will roll her sleeves up and start laying bricks).
  • The engineering and manufacturing approach, or process, is by and large pretty well understood and has been refined over a long period of time (sometimes hundreds of years). The approaches can be taught and are an integral part of the role of being an architect or aero-engineer. Further, these approaches are built around a common language which is also taught and well understood by its practitioners.

A rigorous approach to the field of software architecture needs to recognise the differences whilst at the same time understanding its constraints and build a solid, engineering based approach to its development. This should include killing off the endless debates of agile versus waterfall or structured versus object-oriented and any of the other interminable ‘religious wars’ that we seem to love embarking on and focus in on what matters: applying IT in a reasoned and structured way to solving real-world (and sometimes complex) business problems.

As we enter the new year lets celebrate the field of software development for what it is and help forge the right amount of rigor and discipline in creating a ‘proper’ profession that finally loses the shackles of all those other industries. After all as this guy says (far more eloquently than me) “the processor is an expression of human potential” and is “akin to a painter’s blank canvas” (see this great drawing). I’d like to think of us architects as the painters ready to fill that canvas with great art. Oh heck, but that means we are now comparing architecture with art and that would never do.

10 Things I (Should Have) Learned in (IT) Architecture School

Peter Cripps architecture , , , ,

Inspired by this book I discovered in the Tate Modern book shop this week I don’t (yet) have 101 things I can claim I should have learned in IT Architecture School but this would certainly be my 10 things:

  1. The best architectures are full of patterns. This from Grady Booch. Whilst there is an increasing  need to be innovative in the architectures we create we also need to learn from what has gone before. Basing architectures on well-tried and tested patterns is one way of doing this.
  2. Projects that develop IT systems rarely fail for technical reasons. In this report the reasons for IT project failures are cited and practically all of them are because of human (communication) failures rather than real technical challenges. Learning point: effective IT architects need to have soft (people skills) as well as hard (technical skills). See my thoughts on this here.
  3. The best architecture documentation contains multiple viewpoints. There is no single viewpoint that adequately describes an architecture. Canny architects know this and use viewpoint frameworks to organise and categorise these various viewpoints. Here’s a paper myself and some IBM colleagues wrote a while ago describing one such viewpoint framework. You can also find out much more about this in the book I wrote with Peter Eeles last year.
  4. All architecture is design but not all design is architecture. Also from Grady. This is a tricky one and alludes to the thorny issue of “what is architecture” and “what is design”. The point is that the best practice of design (separation of concerns, design by contract, identification of clear component responsibilities etc) is also the practice of good architecture how architectures focus is on the significant elements that drive the overall shape of the system under development. For more on this see here.
  5. A project without a system context diagram is doomed to fail. Quite simply the system context bounds the system (or systems) under development and says what is in scope and what is out. If you don’t do this early you will spend endless hours later on arguing about this. Draw a system context early, get it agreed and print it out at least A2 size and pin it in highly visible places. See here for more discussion on this.
  6. Complex systems may be complicated but complicated systems are not necessarily complex. For more discussion on this topic see my blog entry here.
  7. Use architectural blueprints for building systems but use architectural drawings for communicating about systems. A blueprint is a formal specification of what is to be. This is best created using a formal modeling language such as UML or Archimate. As well as this we also need to be able to communicate our architectures to none or at least semi-literate IT people (often the people who hold the purse). Such communications are better done using drawings, not created using formal modeling tools but done with drawing tools. It’s worth knowing the difference and when to use each.
  8. Make the process fit the project, not the other way around. I’m all for having a ‘proper’ software delivery life-cycle (SDLC) but the first thing I do when deploying one on a project is customise it to my own purposes. In software development as in gentleman’s suits there is no “one size fits all”. Just like you might think you can pick up a suit at Marks and Spencers that perfectly fits you can’t. You also cannot take an off-the-shelf SDLC that perfectly fits your project. Make sure you customise it so it does fit.
  9. Success causes more problems than failure.This comes from Clay Shirky’s new book Cognitive Surplus. See this link at TED for Clay’s presentation on this topic. You should also check this out to see why organisations learn more from failure than success. The point here is that you can analyse a problem to death and not move forward until you think you have covered every base but you will always find some problem or another you didn’t expect. Although you might (initially) have to address more problems by not doing too much up front analysis in the long run you are probably going to be better off. Shipping early and benefitting from real user experience will inevitably mean you have more problems but you will learn more from these than trying to build the ‘perfect’ solution but running the risk of never sipping anything.
  10. Knowing how to present an architecture is as important as knowing how to create one. Although this is last, it’s probably the most important lesson you will learn. Producing good presentations that describe an architecture, that are targeted appropriately at stakeholders, is probably as important as the architecture itself. For more on this see here.
Enhanced by Zemanta

Evolutionary Systems Development Process

Peter Cripps architecture ,

As I’m now into my second year I reckon I can become more experimental in my blogs. This entry is therefore completely off the scale and will either not go any further or be the start of something much bigger. Comments suggesting which it is are welcome.If we want to be able to build real complex systems (see previous entry for what these are) what might we do differently to ensure we allow truly emergent properties to appear? Is there a development process that we could adopt that both ensured something was delivered in a timely fashion but that also allowed sufficient ‘flex’ such that emergent behaviour was not prevented from happening? In other words is it possible to allow systems to evolve in a managed way where those properties that we value are allowed to grow and thrive but those that are of no use can be prevented from appearing or stopped early on?

Here are some properties I think any such a development process (an evolutionary systems development process) should have if we are to build truly complex systems

  1. The process needs to be independent of any project. Normally we use processes to derive some kind of work breakdown structure which is used to plan the project that will deliver the system. In other words there is a well defined start and end point with clear steps stating who does what by when in between. However for a complex system there is by definition no stopping point; just a number of evolutions where each introduces new (and possible unpredictable) behaviour.
  2. The process should be driven not so much by what users want but by what users do. Most processes start with a list of requirements, possibly expressed as use cases, which describe scenarios for how the system will be used. The problem with this approach is that use cases only describe what a user can envisage doing with the system and will not capture what they actually end up doing with the system. In other words the system is deemed to be ‘complete’ once all the use cases have been realised. However what if the act of delivering the system itself results in new use cases emerging. Ones that were not planned for? How do they get realised?
  3. The process must be both flexible enough to allow users to experiment and envisage doing new things whilst at the same time being robust enough not to allow negative emergent behaviour that will prevent any system being delivered or lead to the system deteriorating over time.
  4. If new behaviour is to be adopted this must be of overall benefit to the majority and must still meet (a possibly changed) set of business objectives. The process must therefore allow for some kind of voting system where the majorities behaviour is allowed to dominate. The trick is not allow new and innovative behaviour to be crushed early on.
  5. The underlying rules that control how the systems responds to external stimuli must themselves be easily adaptable. The process must therefore treat as peers the rules that govern (allowable) behaviour together with the description of the behaviour itself. Rules set some kind of constraint on what is allowable and not allowable by the system.

Of course not all systems that we build will or should be complex ones. As noted previously safety-critical systems need to behave in entirely predictable ways and users should not be allowed to change the behaviour of these where lives could be put at risk. So what type of systems might benefit from an evolutionary approach to development?

  1. Social networking systems where the users are intimately bound up with the system itself.
  2. Commerce systems where the buying behaviour of users might change how and when certain products are sold and at what price.
  3. Financial systems where money markets may affect what users do and how they respond.
  4. Government systems where responses to new legislation or the behaviour of citizens needs fast responses.
  5. Other, you decide?

My next step is to think about what the elements (process phases, work products etc) of an evolutionary systems development process might be.

Complex Systems versus Complicated Systems

Peter Cripps architecture , , ,

Brian Kernighan,co-author of the C programming language, once said:

Controlling complexity is the essence of computer programming.

We live in a world where the systems that we use or come across in our day to day lives seem to be ever more complicated. The flight software system that controls a modern aircraft like the new Boeing 787 “Dreamliner” are, on the face of it at least, almost unimaginatively complicated, simply because of their sheer size. According to NASA the software that runs the 787 is almost seven million lines of Ada code, triple that of the 777. The F35 Joint Strike Fighter has 19 million lines of C and C++ code! Does all this mean that these systems are also complex however and if not what’s the difference? Inevitably there are a large number of definitions of exactly what a complex system is however they all seem to agree on a few common things:

  1. They are made up of a collection of components.
  2. The components interact with each other.
  3. These interactions can result in emergent behavior.

Emergent behavior refers to the property that a collection of simple components may exhibit when the interactions between them result in new, and sometimes unpredictable, behavior that none of the components exhibit individually. So whilst complicated systems may be complex (and exhibit emergent properties) it does not follow that complex systems have to be complicated. In fact relatively simple systems may exhibit emergent properties.

In his book Emergence Steven Johnson gives several examples of the emergent properties of all sorts of systems from colonies of ants through to software systems. Whilst some software systems clearly thrive on being complex systems where emergent behavior is a positive benefit (I’m thinking Facebook, World of Warcraft and SecondLife for example) we might be a bit more dubious about getting on an aircraft whose flight software system exhibits emergent behavior! My guess is that most of us would prefer it if that software showed entirely predictable behavior.

Here’s the thing however. Should it not be possible to build some of our business systems where emergent behavior could be allowed and for the systems themselves to be self adjusting to take advantage of that behavior? How might we do that and what software delivery lifecycles (SDLC) might we adopt to allow that to happen? The interesting thing about SDLCs of course is that almost by definition they build predictability into the systems that are developed. We want those systems to behave in the way the users want (i.e. we want them to “meet the requirements”), not in unpredictable ways. However the thing about systems like Facebook is that the users are an integral part of that system and they drive its behavior in new and interesting ways. The developers of Facebook are able to observe this and respond to new ways in which their system is being used, adapting it accordingly. Facebook has clearly been architected in a way that allows this to happen (Nice one Zuck). The problem with many of our business systems is that they are too rigid and inflexible and do not allow emergent behavior. Even the promise of Service Oriented Architecture (SOA) where we were supposed to be able to reconfigure our business processes almost at will by combining services together in new and interesting ways has not really delivered on that promise. I think that this is for two reasons:

  1. We have failed to adapt our (SDLC) processes to take into account that property and are instead building the same old rigid systems out of slightly new sets of moveable parts.
  2. We fail to recognise that the best complex systems have people in them as an integral part of their makeup and that it is often the combination of people and technology that drive new ways of using systems and therefore emergent properties.

Building and managing complex systems needs to recognise the fact that the same old processes (SOPs) may no longer work and what new processes we new develop need to better account for people being an integral part of the system as it is used and evolves. The potential for emergent behavior needs not only to be allowed for but to be positively encouraged in certain types of system.

 

How to Create Artitecture

Peter Cripps architecture , , ,

Forgive me for pushing yesterdays entry a bit further but I really like the idea of creating architectures that come from more of a right-brained way of thinking. So how should artitects go about creating an artitecture?

  1. Thrash early. Thrashing is a term used to describe the creative brainstorming process that happens during a project. Seth Godin in his book “Linchpin – Are You Indispensable?” says that amateurs thrash late whereas as professional thrash early. Late thrashing introduces bugs which are better to identify early rather than later.
  2. Make mistakes and learn from them. Like Fred Brooks said: “plan to throw one away; you will anyhow”. Use prototypes to understand how new and technically challenging components work but treat these as throwaways not release 1.0!
  3. Deliver (ship) something. According to Steve Jobs “real artists ship”. Delivering something (anything) on time and within budget is one of the great challenges of software development. Time or money (or both) usually run out before anything is delivered. Here’s a different way of looking at it though. Why not see the time/money constraint as a positive rather than a negative aspect of a project? So, if you have to produce something on time and within budget it’s quite simple really. Just work until the time/money run out then deliver something.
  4. Read the rule-book, but then change it. For software development projects the “rule book” is usually the process that is meant to be followed. However it’s important to recognise that there is no “one size fits all” when it comes to SDLC (software delivery lifecycles). SDLCs are good but don’t follow them too slavishly. Be ready and prepared to customise them to your needs. A good SDLC will have this flexibility.
  5. Seek forgiveness not permission. I got this from an ex-colleague. In many companies today you are overwhelmed by (often) petty rules of engagement. If you always follow the rules that are supposedly “needed” to get a piece of work done then you won’t deliver. You’ll have done your job (followed the rules) but won’t have done the work that your client wants. Better is to not follow every rule and ask permission for doing something but to just do it and ask for forgiveness when the rules get broken (unless those rules are there to keep you out of prison of course).

Okay here’s the real irony in the above. If you read and believe my point number two from yesterday (artitects don’t follow the process in the manual, instead they write the manual) you won’t be following any of the above; instead you’ll be creating your own way of doing artitecture.

Architecture Refuseniks

Peter Cripps architecture ,

re-fuse-nik (noun) a person who refuses to cooperate with a system or comply with a law because of a moral conviction

Some IT architecture refusenik types who may or may not have moral convictions about this sort of thing:

  1. It’s all in the code – Just wants to get on with cranking out what really matters (to them) the code. Thinks that anything to do with architecture is poncey nonsense and that all architects need to “get real” and realise that code is king.
  2. It’s all in the process – Can create the worlds most complex process that will deliver, er well nothing actually because no one will ever take any notice of it and just get on with the real work anyway.
  3. It’s all in the clouds – Will pontificate endlessly on things like ‘governance’, ‘process’, ‘reviews’ but will never actually deliver anything.
  4. It’s all in the model – Opposite of the coder. Thinks the system is completely describable by models and says their work is done, and the system is ‘complete’ once 35 (or more) UML models have been created.
  5. It’s all in the plan – As long as we have a plan that is at least 8 levels deep and has every minute of every day accounted for it’ll all just work at the end of it.