Abi James

December 2017

Usability evaluation practical

marine_blue _logo
5 is the magic number!
Nielsen and Landauer (1993) found that the likelihood that at least one participant will encounter the problem during usability testing is on average p=0.31
Based on that, 5 users would be expected to find 85% of the usability problems available for discovery in that test iteration.
But in real-life not everyone undertakes the same task and users are all different!

How many people need to be involved with usability testing?

  • Allow practitioners to report results in finer detail with more evidence than personal judgment.
  • Standardization is key to generalizing a finding from a sample to the greater population.
  • easier for researchers to communicate findings when referring to standardized metrics.
  • easy to compare different design iterations throughout the development process and to make comparisons between sites.

Benefits of standardized usability questionnaires are:

  • What areas do you want to evaluate? Length of questionnaire - one questions to many
  • How concerned about reliability are you?
  • How are you going to disseminate it? How many responses do you need?
  • Cost! Open vs commercial
  • Localisation

Usability Surveys – which one to choose?

No of questions
Software Usability Measurement Inventory (SUMI)
Users’ perception of the Efficiency, Affect, Helpfulness, Control and Learnability of a system
~$700 a month
Post-Study Usability Questionnaire (PSSUQ)
users’ perceived satisfaction with a product or system
Software Usability Scale (SUS)
measure users’ perceived usability and learnability of a product or system
Used in over 40% of published usability studies

Comparing Standardised Questionnaires https://measuringu.com/ux-questions

Think aloud protocol:
  • ask test participants to use the system while continuously thinking out loud—verbalising their thoughts as they move through the user interface – and being observed
Support through evidence from task and usability surveys:
  • Task questions measure ability of user to complete defined task. Can be 1 – 5 questions
  • Usability questions

Combining qualitative and quantitative testing:
Think Aloud protocol plus survey

  1. Define the tasks you want to test
  2. Define the users you want to include in your tests
  3. Define the process for testing and ethical considerations
  4. Recruit your participants
  5. Under take usability test and observer the users
  6. Gather quantifiable data to back up observations
  7. Analyse the data and report

Usability testing protocol

Working in groups of 2 or 3:
  • One person take on role of observer, others the user
  • Complete a task on SlideWiki, thinking aloud as you go
  • Observer should make note of positive and negative comments made by the user
  •  Switch roles and complete another task
  • All complete the survey:


Consists of
  • Background information on the user
  • Prior experience with similar tools
  • Software Usability Scale (positive version)
  • 3 questions from the NASA task index
  • Free text questions

SlideWiki Evaluation Survey


Questions and comments