This recent blog entry from Seth Godin on “the culture of testing” set me thinking about software architecture and testing. Is it possible to ‘over test’ applications or systems? Is there a point at which you need to stop testing and let your software ‘go free’ so users can ‘complete’ testing themselves? Let’s be clear here:

• Software needs testing very rigorously. No one wants to fly on an airplane where the on-board flight control software has not been fully tested.
• I’m also not talking about the well established practice of releasing beta versions of software where you get together a bunch of early adopters to complete testing for you and iron out the “last few bugs”.
• Testing software against known requirements is most definitely a good thing to do and I’m not advocating just running your system tests against a subset of those functional and non-functional requirements.

Where it gets interesting is when you don’t overly constrain the architecture so that it allows users to take resulting application or system and evolve it (test it) in new and interesting ways. When defining an architecture we usually talk about it having to address functional as well as non-functional requirements but there is a third, often overlooked, class of requirement referred to as change cases or future requirements. Change cases are used to describe new potential requirements for a system or modifications to existing requirements. Change cases usually come from the specifier of the system (e.g. “in three years time we want introduce a loyalty scheme for our regular guests”). Some change cases however are not envisaged in advance and it’s only when users of the application get hold of it that they explore and find new ways of using the application that may not have originally been thought of. Such applications need to be carefully architected, and tested, such that these change cases can be discovered without, of course, breaking the application altogether.

So, by all means ship systems or application that do what they say on the tin but also lets users do other, possibly more interesting things, with them that may lead to new and innovative uses that the original specifiers of the software had not thought about.