• Calling All C++ & AI Game...
• SAS Analyst
• Program Manager - Long Term...
• Technical Lead
• Wintel Systems Specialist

Usability testing: Is it really worth it?

John Eklund, Information Age

19/10/2006 20:54:05

What is usability testing? The objective of usability testing is to make computer systems easier to use, allowing users to undertake their tasks in a manner that is effective, efficient and satisfying. Usability as a goal is achieved when, "the people who use the product can do so quickly and easily to accomplish their tasks". (Dumas and Redish, 1999). Usability focuses on the interaction between user and system, which typically takes place in some form of interface.

To understand how it is done, it's necessary to look more broadly at notions of Human Computer Interaction (HCI). There are four core elements in any human computer interaction: the user, The tasks the user has to perform, the system (software and resources) that enable them to perform it, and the context in which it is performed.

These four elements or variables together define the environment in which the particular HCI takes place.

The aim of a usability testing is to provide feedback on system designs from a human factors perspective to improve those designs. This is done by creating a modelled environment, and attempting to replicate the elements of user, tasks and context while examining the system variable.

In other words, we fix the user, tasks and context variables and examine the system, and measure its performance in a qualitative sense. The output of a usability evaluation is a description of how various characteristics of the system perform, and how they might be altered to improve the performance of the HCI interaction in the environment.

Usability is a measure of the quality of the system as it is used. It is a component of the broader set of criteria termed Quality in Use (ISO 14598-1).

Why we need usability testing Usability testing as an activity is an important component of User-Centred Design, an overarching methodology that involves consulting users at various stages of the software development lifecycle, typically through gathering user requirements, testing visual designs and partial prototypes, and validating final designs against user expectations and tasks.

Usability testing determines the extent to which a design meets user requirements, whether they are explicit or not

Usability testing is necessary, because even if user requirements are very carefully gathered and documented at the start of a development project, the manner in which those requirements are expressed in an interface design, and a user group's reaction to that design, can both vary significantly, affecting both user perceptions and usage of the application or Web site.

This can cause a lot of surprise and angst for the developers of the system. Fundamentally, developers of software systems are too close to a project to be able to objectively assess whether what they have created is a faithful interpretation of requirements, or one that is likely to meet the needs of the users.

The independence of testing is paramount - you just can't test your own designs

The methods of usability testing Usability testing can be undertaken through a variety of methods such as questionnaires, focus groups, paper prototype walkthroughs, task-based functional testing, eyetracking, expert reviews and heuristic evaluation. There are many others and an infinite array of variants on approaches, and often it is by a combination of methods that most efficiently attains the testing outcome.

All these methods hope to achieve one goal: to assess the quality of designs against a range of agreed standards and make recommendations for improvements. Those "standards" are selected at the start of the test process, and form part of defining the test objectives. Standards can be what users expect, what they like, how they perform on task completion, as well as rules for good interface design and best practice. An iterative software development process involves specifying what is to be built, creating successively more tangible artifacts that represent that specification, and evaluating them from multiple perspectives.

Each evaluation gauges software quality and results in a modification to the specifications before entering the next build phase. The number of iterations depends on project attributes such as business goals, size, budget and uniqueness, the skills of the development team, and the adopted development methodology.

One of the most common errors is to leave usability testing too late. It should be introduced quite early in the development lifecycle, typically at the point where an early partly functioning prototype is developed. Earlier testing is more exploratory when changes to designs are acceptable. The later the testing, the more it will become a validation exercise that focuses on identifying problems with minor interface elements. Major changes become more expensive and difficult to implement.

When usability testing is done properly, at the right time, with an appropriate scope, and a carefully expressed set of test objectives, it will add direct value to the development process, identifying issues and making recommendations for improvements. It provides the development team with a set of actions:

Items in an interface to fix, edit or implement, as well as an objective assessment of the quality (i.e. quality in use) of what has been created so far. As a validation exercise, it provides project sponsors with assurance before launch that the developed system will be usable by the target market and support the business goals of the sponsors.

But, do I really need to do usability? So, if you are one of those pragmatic project manager types in charge of the budget of an applications or Web site development project, I can guess what you might be thinking: you probably already know user-centred design is good, and that usability testing is an important part of it, but wonder if it is worth the cost of hiring an external party to do it.

You may believe that your team is taking into account usability principles while developing. And, while you would like your users to have a good experience with your application or Web site, you don't necessarily want to spend an already stretched budget just to make your users happier. You might wonder if there some quantifiable benefit from usability testing from a business perspective.

Obviously usability testing is going to cost - it's a specialist activity completed by consultants. The question is, what return on the cost can you expect if you invest in independent usability testing as part of a commitment to user centred design? In order to answer this, we can revisit at a few commonly held misconceptions about usability testing:

• Usability is obvious, my development team is experienced and know their users very well • Usability testing is like paying for someone to be critical rather than getting on with creating usable designs in the first place • Usability testing is just another process that will impact delivery timelines • Usability is not related to the business goals of the system, only how nice the system is to use.

I need to acknowledge that all of these concerns can be real and will occur if the testing is not well-integrated into the development lifecycle or if the testing is poorly scoped or implemented. However, carefully planned testing can adequately address them all - and easily.

It is the relationship between usability testing and the business goals of the system that I would like to focus on in the remainder of this article, using a case study from work undertaken by Access Testing last year. We look first at the potential benefits for usability testing (see Bias & Mayhew, 2005 for a thorough analysis), and see how these worked in the example.

Potential benefits First, there are potential business benefits in developing the product, by: • Producing a product that has only relevant functionality • Detecting and fixing usability problems early • Reducing amount of major redesign • Minimising documentation • Maximising product design to work well not just look good • Reducing the risk of project failure.

Second there are potentially direct business benefits. In particular e-commerce sites can be improved by increasing the number of customers who will: • Find products they want • Find information easily • Be satisfied and make repeat purchases • Trust the site • Not require support • Recommend the site to others • Support and increase sales by other channels.

Product sales can be increased as a result of the usability of the product, by: • Improving the competitive edge by marketing the product as easier to use • Increasing numbers of satisfied customers • Obtaining higher ratings in product reviews.

Employers can benefit from easier to use systems, through: • Faster learning and better retention • Reducing task time and increasing productivity • Reducing errors that require correction • Reducing staff turnover as a result of higher motivation and satisfaction • Reducing time spent by other staff providing assistance when users encounter difficulties.

Suppliers can also benefit from: • Reduced maintenance costs • Reducing support and help line costs • Reducing training costs.

A case study Last year Access Testing completed comprehensive usability testing for well-known insurer AAMI on an online system that allows users to locate insurance products, purchase and manage their own policy details online.

As more businesses increasingly rely on online transactions to deal directly with their customers, the e-business division of AAMI has a goal to position themselves as best of breed in the insurance industry. E-business generates a significant percentage of the total sales of insurance for AAMI and is growing at a rate of 30 per cent per annum.

AAMI's e-business ensures innovation and processes are mirrored in AAMI's leadership and reputation. AAMI introduced automated online quoting and payments in 2001 and this is now available for car, home and compulsory third party.

A recent development, My Policy Manager, allows customers to view their policies online and soon, conduct a range of maintenance and transactional functions that include renewal payment, changing addresses, cars, drivers, altering modifications and accessories and much more.

Allowing customers to manage their own accounts places control back to the customer, but also reduces expensive call centre costs. Online transactions cost a fraction of one completed over the phone.

The cost and the benefit goals AAMI's online system is a part of a business unit that is delivering $90m per annum. These systems are increasingly taking over call centre activity, allowing customers to manage their own insurance needs. It is estimated that these systems deliver 10 per cent of revenue and this share will increase with future enhancements.

The testing took three weeks and cost a total of $25,000. It is calculated that the after tax cost will be recouped in the first year if there is a 1 per cent improvement in the "efficiency" of the online system.

The usability testing In accordance with AAMI's customer-centred focus and knowing that even small improvements in the interface can lead to substantial savings, AAMI commissioned Access to conduct usability testing, including eyetracking, on the My Policy Manager Web site. Twelve individual 90-minute test sessions with representatives of the target market were undertaken including questionnaires, interviews and observation of tasks.

Eyetracking analysis was conducted from the AAMI home page to the My Policy Manager area. eyetracking is a technology that follows a user's eye movements on screen with the aid of an invisible infra-red beam that fixes on the eye by means of a camera. It also captures all keystrokes and mouse movements to build a composite profile of visual activity. The testing was done in Access Testing's usability labs in Sydney.

Usability issues found The resulting report identified several usability problems with the existing designs, even though many were relatively minor.

One noticeable problem related to the layout of the My Policy Manager entry area, which had a number of clickable links with the login button at the bottom of the page. Many users selected the "Take Tour" Distracter, which was positioned close to the Key Text "View, Manage and Renew your own policies online". Users had difficulty navigating their way back from the tour to the key transactional area.

Eyetracking on this area confirmed that the issue was not the terminology of the button "Log-in", but users actually did not see Log-in before selecting one of the other links.

Of the many issues uncovered in the designs, this is one example where users said they might "just give up and make a phone call". This was even before users started.

Eyetracking provides quantitative data to support and validate usability findings.

Following the study, Access Testing recommendations were implemented in a new round of design changes. The revised design has a simplified page layout, with fewer calls to action.

Other issues identified in the round of user testing related to the positioning and labelling of buttons, navigation design, and the amount of feedback the Web site provided users at the completion of a process. All of these issues (there many identified) are reported in terms of location, cause and severity and structured recommendations were provided to remediate them.

Business benefits The benefits to AAMI in undertaking user testing on My Policy Manager can be measured in increased online signup completions exceeding planned projections, as well as customer satisfaction in using the Web site, promoting positive views of the organisation among its customers.

(AAMI recently won an award for their Web site - The award presented at the Australian Banking and Finance magazine award night was for "Best Australian Insurance Website". The award recognized the transparency the Web site gave AAMI processes and the ease of use for customers.

The benefits of the redesign following testing were: • The development team had clear directions for redesign, saving effort • Project sponsors obtained a view of how customers would react to the site • The insurer has been nominated most usable insurance Web site, which was considered a tremendous promotional boost • The revenue generated by the sign up process on the site increased • Call centre activity, and the number of operators needed, decreased despite an increase in revenue • The appearance and ease of use of the site improved • The average amount of time users spent on the site before purchase decreased • The number of drop-outs decreased.

Conclusion In this article I have attempted to make a case for independent usability testing as a valuable part of a user-centred development process, particularly when conducted early in the development lifecycle. The AAMI case study shows us the benefits from a number of perspectives, all leading to increased revenue for the company. These benefits will be easier to quantify on any project if before and after measures are put in place at the start of the project.

References Bias R & Mayhew D (2005) Cost-Justifying Usability - An Update for the Internet Age. Morgan Kaufmann, CA.

Dumas, J & Redish, J (1999) A practical guide to usability testing. 2nd Ed. Intellect.

Dr John Eklund is a senior usability consultant and e-learning specialist at Access Testing.