• Project Manager
• Senior Desktop...
• 1st / 2nd level Help Desk -...
• IT and Projects Manager -...
• IT MANAGED SERVICES - BDM -...

Standards need more rigour

Tom McBride, Information Age

12/10/2004 13:47:58

Additionally changing a standard can incur significant costs to the user which puts pressure on the developers to get it right in the first place, but also to seriously consider the consequences of changes. Like software, the quality level needs to improve. But also like software, this won't happen using the same development methods and, instead, requires different methods. In other fields, like software development, methods of achieving the same quality objectives have included field trials and more rigorous methods of expression and checking. Like software, higher quality levels are seldom achieved by pressuring those who are already doing the best they can under the circumstances. It requires different methods. Standards are currently developed by writing narrative English, then sending it out to a wide range of people for critical review. The problems with this are that it is difficult to achieve consistency across long documents or between documents and, although circulation is wide, very few people are knowledgeable enough or take the time to do more than a cursory review. Two hundred pages of standard is a daunting thing to read and hardly bedtime entertainment. At its best peer review is an excellent flexible system of checking the quality of a standard but at its worst it does not ensure the quality levels expected and required of current-day standards. But it is the most common system of review when the subject matter is diverse and involves concepts rather than numbers.

Some standards have great commercial significance. ISO 12207, ISO 15504, CMMI and ISO 14143 are examples of standards with a reasonable sized user base. Any changes to those standards, no matter how well justified and no matter how much better it makes the standard, still incurs upgrade costs. In Australia we tend not to think too much about the costs of changing a software development methodology because it doesn't seem to be that much of a bother to read what the new methodology says and follow it from now on. But Australia has less appreciation of large scale than do Japanese companies, American companies and European companies. Some years ago, when some seemingly small changes were being proposed to ISO 12207, the Japanese delegation quietly expressed some concern.

It seems that Japan had adopted ISO 12207 nationally and any changes to it would incur significant costs to Japan as a whole. Similarly when the Software Engineering Institute wanted to publish the Integrated Capability Maturity Model (CMMI) and withdraw its predecessor, CMM, a large number of organisations simply refused to convert to the new model. Having spent significant money on getting the development processes required by CMM installed and everyone trained in their use, they weren't about to happily spend the same amount all over again. So CMM has been maintained for an interim period. During its development there has been considerable difficulty gaining consensus on ISO 14143 - Functional Size Measurement. One of the reasons, but not the only one, is because the existing base of some 80,000 organizations already use the IFPUG method to measure the functional size of their systems. Any change to the method raises problems with the validity of the existing database of measurements. What happens to all that data? What happens to the investment in the training in IFPUG alone, never mind anything that actually uses the measured size to, for example, estimate the size of a project? What of contracts drawn up based on IFPUG measures? Are they still valid? And, of course, we know about the costs of upgrading the entire QA system every time ISO 9001 is revised, which is about every five years.

Change is inevitable, we know and acknowledge. But it would be nice if the standards were well developed and rigorous in the first place so that the passing of time brings on necessary changes rather than bug fixes. It is annoying to be faced with changes to standards, and the cost of conforming to those changes, when the standard wasn't as well thought out as it should have been.

When consistency matters, as it does in the process standards of ISO 12207, ISO 15288 and ISO 15504 rather than writing the standard as a text document they could be written as a database. Databases are good tools with which to ensure consistency. This was done as an experiment on one standard recently with significant effect on the level of consistency we were able to achieve. Unfortunately the idea of using a database proved to be a little too radical for some and it wasn't carried through although the resulting consistent clauses were. As it happens when we looked into it, a database is an acceptable way to express a standard. Maybe next time.

Review by domain experts will catch many flaws of clarity, ambiguity and consistency but won't be able to check usability. That would require a trial.

We are all very familiar with clinical trials and expect that many consumer goods will be thoroughly tested before becoming available to consumers. We wouldn't accept the view that knowledgeable people have reviewed the specifications and looked at the final product and they think it will all be OK. Yet this is what we do with standards. There is seldom a trial to see if the standard does achieve what it sets out to do, like describe activities that will actually result in a good specification or good design. There is seldom a trial to see if the intended consumers can understand the thing. Nothing to check how easy or costly it is to implement. Nothing to check how it affects or interacts with other activities the consumer might be involved in.

One of the software engineering standards was trialled as part of its development. ISO 15504, Software Process Improvement and Capability dEtermination, was published first as a technical report so that there was a two-year period, instead of the normal five-year period, in which it could be trialled throughout the world and the experience of those trials fed into the subsequent revision. Full clinical trials are expensive and standards don't need to achieve the same level of proof that they will do no harm. Instead it would be better to begin with the objective of proving minimal levels of usability. Rather than pitching standards development and trials as requiring the best available talent, perhaps it would be better to look at what would be "good enough" to achieve a minimal level of usability and to begin the expectation that standards should be trialled. We first need to establish a general method of conducting standards trials.

But this is not a very charitable time. Few companies are likely to generously say to one of their most valuable people, for domain experts tend to be valuable, "Here, take a few hours away from this project, on which the future of the company hangs, and tell me whether you think this draft standard makes sense." There is no money in it for them, and precious little publicity. Nor is the academic community likely to spend their precious time developing a research program to trial a standard when work on the standard doesn't count toward any research quantum. It is unreasonable to expect the standards developers to be aware of everything. The people who develop standards are all very able people who volunteer their time to a good cause. Their only payment is the same bragging rights that go along with any other volunteer work. But there aren't that many of them and they all have limits on how much time and energy they can devote to the task.

Perhaps there are some who are domain experts and could afford the time to review a standard and send in some observations and suggestions for improvement, And perhaps some companies really could use the standard, assuming it was well founded and actually helpful. Perhaps they need it enough that they would be prepared to trial it so long as they got some help and so long as it didn't cost them real cash. That would be a start. Tom McBride Chairman, ACS National Standards Committee