Usability Testing vs. User Acceptance Testing: Understanding the Differences

It doesn’t matter what you call it. Here’s the bottom line: you need to test software with its future users.

One of the topics that create much confusion in the software world is the difference between usability testing (UT) and user acceptance testing (UAT). User experience (UX) professionals only worry about usability testing, while developers and Q&A people focus on user acceptance testing. Other stakeholders use both terms interchangeably believing in the sentiment of, “It doesn’t matter what you call it — as long as we test our software with users.”

In fact, both tests involve end users utilizing software with the goal of finding shortcomings. However, the flaws being identified differ, and with that the purpose of why you want to execute each test. Further, the point in time when the tests are conducted may also differ depending on the development process that is being used. So let us try to untangle all of this.

What Do the Tests Do?

In both usability testing and user acceptance testing end users engage with a product and go through certain test scenarios. Observing test users succeed or fail with test tasks and hearing their comments about the product itself provides insights into the quality of the product.

UT is concerned with understanding the user experience that manifests itself when a user engages with a product or concept. UX is a comprehensive concept describing and measuring the objective and subjective effectiveness and efficiency of the interaction — that is to what degree users can achieve objectives and how much effort they must put forth.

UX also includes the psychological impact of the user-product interaction. Is it perceived as comfortable, joyful, stressful, confusing or straight-forward? During usability tests qualitative and quantitative aspects are being assessed. Qualitative aspects encompass the sentiment of test users; their comments and reactions about the product. An example is a finding “Test user is surprised by the ‘Close’ button saving their settings in the modal window.” Quantitative aspects are measurable findings, like task success rates, task completion times, number of user errors committed during task execution, etc. For instance, a finding may be: “85% of test users completed test task #1 in under 1 minute.”

UAT is concerned with understanding if the product does what it is supposed to be doing and that its target audience in fact can achieve their objectives. In other words: UAT checks whether defined business requirements are being met by the product. This means that the right functions are built in, and that the code is faultless.

For an example, we may test whether an ecommerce website allows users to add an item from a product detail page to the shopping cart. The product passes the test if pre-determined acceptance criteria are met. For example: User selects the desired quantity of the product; user clicks the “add to cart” button; page shows a pop-up message “added to cart.” UAT considers usability issues that test users comment on, yet the usability itself is not being measured.

Usability testing

Does the product produce a good user experience?

  • Does it allow users to effectively, efficiently and comfortably achieve certain objectives?


User acceptance testing

Is the product fit for purpose?

  • Does it enable users to achieve certain objectives?


Related Article: UX Is a Continuous Investment for Profitable Companies. Here’s Why

What Value Do the Tests Provide?

As mentioned above, seeing how actual users from the target demographics interact with the product provides an impartial and authentic understanding of the product’s quality.

In most cases usability testing is used formatively — it entails a rework of the product to mitigate identified shortcomings. The testing is part of the iterative design and development process and ensures that usability defects are uncovered as early as possible and fixed before the product is released to the market.

The role of user acceptance testing on the other hand is to verify that the product to be released serves its stated purpose, and that the code is faultless, thus allowing target users to successfully work with it. It provides the company that builds the product a level of assurance that what they are about to release is delivering the capabilities that it was built to provide. Traditionally, this verification has happened only once and at the end of the development process — a summative testing approach oftentimes called “beta test.” As I will explain next, this has changed in the age of agile development processes.

When to Carry Out the Tests?

The type of development process being used to create the product has an impact on the timing of the tests. Let’s consider waterfall and agile processes.

Waterfall Process

To help shape further design/development iterations, UT in a waterfall process, where each of the development phases (analysis, design, implementation and testing) are done one after the other, does NOT happen during the regular testing phase. It happens instead already during the design and development phases. Because the cost of change grows exponentially during the product development process, the sooner UT can be carried out, the better. All that is needed is something experienceable by a test user (more about that later).

UAT in waterfall is conducted at the end of the development process. It is not done to improve the product further, but instead to verify that the business requirements have been met and the code is fully functional to enable users to achieve their objectives. If the test reveals that this is not the case, then this poses a challenge, because the developer must then go back in the process (to stay in the waterfall metaphor: swimming upstream) to mitigate the issues in earlier process stages — which is costly in terms of time and resources.

Agile Process

In agile, where a development project is sliced into micro-projects called sprints, for each product development phase (analysis, design, implementation, and testing), both UT and UAT are principally part of the individual sprints. In practice, with sprints typically being only 2 weeks long, it can be challenging to fit the tests with end users into the tight schedule. It is an option to carry them out in a certain cadence as own, separate sprints.

Interestingly, when UAT is done within agile, it is formative rather than summative — it no longer happens only once at the end, but throughout the product development. Since both tests are formative, providing insights that are being used to further advance and optimize the product, you could argue that they are essentially one and the same. However, the difference in scope remains: UT focuses solely on the user experience while UAT focuses on adherence to business requirements and flawless code.

Related Article: Agility Is No Longer Optional in Business

Source link

We will be happy to hear your thoughts

Leave a reply

Reset Password
Shopping cart