In-Depth Look at Developing a Standardized Usability Test for Your Product or Website

Credentials

I don’t plan on beginning all of my articles with a bio, but since this is one of my first posts I thought it might be good to provide my credentials, without them you might wonder what qualifies me to write such an article. I’m Ro Dixon, I’ve been a UX Designer for nine years working for enterprises that specialize in applications for Healthcare Providers (I’m also a front end developer and web designer- but that’s not super relevant to this post).

I was given the task of designing a Usability Testing process for a recent employer which would be used across our entire product line, and I want to share what I learned from that experience with you.

Purpose

The purpose of this article is to provide UX Designers and Researchers the following things:

1. A comprehensive, yet concise overview of the Standardized Usability Testing options at their disposal.

2. Details on why I chose the method I did for my own usability needs.

3. Details on how you can take that method and manipulate it to suit your own.

Usability Metrics

Usability metrics refers to usability testing that rates user feedback in the following categories: Efficiency, Effectiveness & Satisfaction.

The photo above was found in Jeff Sauro’s article titled, “A Method to Standardize Usability Metrics Into a Single Score.” In his article he makes a very good point, that the irony of usability metrics is that they “…need to be easier to use”. I wouldn’t exactly say that Sauro’s article is an easy read, but it is very well researched and informative.

To summarize, you need to review the differing methods of usability testing and select one that works for you, your client and your product or website. Once you have chosen a method, it’s time to cater it to your own specific needs. It’s a lot of work up front, but once your process is in place, the rest is a breeze.

Methods of Standardized Usability Testing

Let’s look at some Standardized Usability Testing Methods (in no particular order).

SUPR-Q

The SUPR-Q is an eight question survey that focuses on usability, credibility, loyalty, and appearance. The first seven questions use a likert scale for participants to rate their response from strongly disagree to strongly agree. The eighth question asks how likely the participant is to recommend the website to friends.

PROS

• The good thing about this survey is its length. At only 8 questions it’s quick and easy for participants to take.
• The SUPR-Q offers ten possible questions so you have a little flexibility in which you choose.
• Scoring your results is really simple.

CONS

• You must purchase a license to use the SUPR-Q starting at $3,000 (at the time the article was written) which is pretty steep for most small to medium-sized businesses.
• This survey option also isn’t as applicable to software, saas, or online applications.

If you’d like more information the SUPR-Q website is actually pretty well written. You can also purchase a license from measuringu.com if you decide this survey is right for you.

SUMI: Software Usability Measurement Inventory

The SUMI is an industry standard questionnaire for analyzing the usability of online software and applications. Which means if you are mainly testing websites this is not the survey for you. I won’t spend a ton of time on this one, for one reason, it’s 50 questions long. The SUMI is just too flipping long for a usability survey. And frankly, for those of you asking for unpaid volunteers to take your usability test, you won’t have any willing to take this one.

Additional CONS:

• Many of the questions are repetitive, so I personally just don’t think it provides more value than much shorter methods.
• The SUMI requires an expensive license to use.

If I haven’t deterred you yet and you want to learn more about this method, take a look at these links:

• http://sumi.uxp.ie

PSSUQ: Post-Study System Usability Questionnaire

The PSSUQ is a 19-item survey developed by IBM to use in scenario-based usability testing. IBM’s goal was to gain insight into the following five system usability characteristics.

1. Quick completion of work

2. Ease of learning

3. High-quality documentation and online information

4. Functional adequacy

5. Rapid acquisition of usability experts

Like many other methods, PSSUQ uses the likert scale. Participants rate each question from one to seven, one indicating Strongly Agree and seven, strongly disagree. To calculate the overall score you simply average the responses.

PROS

• Using the PSSUQ is 100% free!
• Questions are pre-written and vetted, which is a time saver.
• At only 19 questions, it’s still short enough that many users will be willing to complete it.
• It’s from a very credible source.

CONS

• There are shorter surveys that provide the same amount of value.
• There isn’t as much information available for this method as others.

If you want to know more, feel free to browse these resources.

• Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies
• www.conetrees.com

QUIS: Questionnaire for User Interaction Satisfaction

The QUIS was created by a team of researchers from the University of Maryland. They claim that

“The QUIS was designed to assess users’ subjective satisfaction with specific aspects of the human-computer interface. The QUIS team successfully addressed the reliability and validity problems found in other satisfaction measures, creating a measure that is highly reliable across many types of interfaces.” (QUIS)

Unfortunately the QUIS official website doesn’t offer any examples, but I did find one here that is 27 questions long.

PROS

• Easy to score
• Easy to prepare and get in the hands of users

CONS

• The QUIS is a little pricey, a commercial license will cost you $750.
• At 27 questions, it’s border-line too long for unpaid volunteers to complete. If you’re planning to pay participants or offer rewards, you might be able to use this survey.
• Without purchasing a license, I’m unsure of the structure (which makes it difficult to know if you want to invest in it or not).

WAMMI (Website Analysis and Measurement Inventory)

The WAMMI is a 20-question survey developed using psychometrics. The makers of the survey claim that it,

“summarizes the essence of User Experience — as experienced by the visitors themselves. The questions allow the visitors to speak with their own voice, rather than having ideas imposed on them. Each question represents a vital aspect of User Experience…”

The survey focuses on five major categories: Attractiveness, Controllability, Efficiency, Helpfulness and Learnability.

Unlike prior surveys discussed here, the WAMMI is administered via the owners of the survey and results are added to a global database which others can reference in the future. WAMMI also will provide metrics comparing your results to others within their database, which is a major selling point for their questionnaire.

PROS

• WAMMI will run the test for you, which could be great for those of you with little resources allotted to usability testing.

CONS

• There is absolutely zero flexibility with the 20 base questions. WAMMI allows you to add questions such as collecting demographic data, or asking open-ended questions, but that’s all the choice you have.
• WAMMI recommends that you receive a minimum of 30 responses. Most other surveys can be completed with as little as five.
• WAMMI customizes their prices to each client, as the more questions you add or languages you request, the more you will pay for their services. Which means I have no idea how much this survey costs, but my assumption would be it isn’t cheap.

Survey Example: http://www.wammi.com/samples/index.html

ASQ: After Scenario Questionnaire

The ASQ is provided to participants directly after they have completed a scenario on a website, online application, or software. The questionnaire is only three questions long, and to score the results you simply find the average all given feedback and calculate the mean. Since a one indicates the participant strongly agrees with the positively phrased statement, the lower the score the better.

Below is an example.

PROS

• The ASQ is the shortest of all the usability surveys currently on the market. A good option for those requesting unpaid volunteers for their feedback.
• ASQ is 100% free.
• Easy to create/share.
• Easy to score.

CONS

• Since the questionnaire is only three questions long, you won’t be able to use it for in depth analysis, or very specific statements surrounding a product or feature.

HONORABLE MENTION

• CSUQ (Computer System Usability Questionnaire), Click here for an example.
• USE (Usefulness, Satisfaction, and Ease of Use)
• ACSI (American Customer Satisfaction Index)
• NPS (Net Promoter Score)
• CxPi (Forrester Customer Experience Index)
• TAM (Technology Acceptance Model)

Finally, we have the SUS (System Usability Scale)

My personal favorite, and my choice for my own usability testing is none other than the System Usability Scale survey. The SUS is ten questions long using the likert scale. This survey was invented by John Brooke in 1986 and has been vetted by researchers for the last 30 years! Probably my favorite thing about this survey is you can cater the questions to suit your own needs, and the scores are still comparable to industry average, which is currently 65–70 depending on which source you ask.

The template questions focus on three main categories of experience:

Satisfaction

• The attractiveness of the product
• The emotional response it provides the user (Is it frustrating?)
• Does the product meet user expectations

Efficiency

• The organization of product elements should be clear, useful & orderly
• Our product plays well with users browser & complementary tools

Effectiveness

• Errors
• Completion rates
• Content
• Sometimes questions in this category align with efficiency also.

If you have any additional categories you may add them, I would try and stick to five or less, because the SUS is only 10 questions in length, if you have more than five categories then you will only be able to ask two questions or less in each category.

For my employer, learnability was a very important factor in our particular products. If users couldn’t learn to navigate and use our online applications quickly, then healthcare providers would need to implement lengthy training courses and/or materials, which would lessen their ROI. Therefore in addition to these three categories we added learnability to the list.

Then we decided to structure our own personal SUS surveys in the following manner. Maintaining consistency across all of our surveys would ensure that our scores could be compared to one another over time.

Of the 10 questions, we’d have the following in each category.

· Satisfaction: 2 Questions

· Learnability: 2 Questions

· Efficiency: 3 Questions

· Effectiveness: 3 Questions

Writing SUS Questions

Here is a question template that usability.gov provided:

1. I think that I would like to use this system frequently.

2. I found the system unnecessarily complex.

3. I thought the system was easy to use.

4. I think that I would need the support of a technical person to be able to use this system.

5. I found the various functions in this system were well integrated.

6. I thought there was too much inconsistency in this system.

7. I would imagine that most people would learn to use this system very quickly.

8. I found the system very cumbersome to use.

9. I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

This is where the SUS becomes a little complex.

You’ll notice that the odd numbered questions are positively phrased.

Example: I like using this product/feature.

Desired Answer: Strongly Agree

The even numbered questions are negatively phrased.

Example: I found the product/feature cumbersome to use.

Desired Answer: Strongly Disagree

It’s important to maintain this structure when you create your survey, if you don’t than the scoring won’t be applicable. With that said, if you really don’t like this structure you could still take the questions, phrase them positively and then just take the average answer in each category. The negative in doing that is you will not be able to compare your score to your competitor’s SUS or the industry average. I’m only bringing it up as an option if you so choose.

Here are some examples within each category.

Satisfaction

· How would you rate your enjoyment using our Product/Feature?

· How comfortable were you with using our Product/Feature?

· This Product/Feature has all the functions and capabilities I expect it to have.

· The Product/Feature meets my expectations.

Efficiency

· The organization of information in the Product/Feature is clear.

· The Product/Feature was simple to use.

· I can’t effectively complete my work using the Product/Feature.

· Whenever I make a mistake using the Product/Feature, I can’t recover easily and quickly.

Effectiveness

· When using the Product/Feature, it is easy to find the information I needed.

· The Product/Feature gives error messages that clearly tell me how to fix problems.

· The Product/Feature has all the features available that I need to perform my job effectively.

Learnability

· It was easy to learn to use the Product/Feature.

· I need more tool tips, hints and guides to use the Product/Feature.

Calculations

Users will give each question a score from 1 to 5. For odd numbered questions you subtract 1 from that score. For even numbered questions subtract the value of their answer from 5. Once you have the SUS for each, take the sum of all and multiply by 2.5. The score should be 100 or less. This is NOT a percent. Below is an example, accompanied by the calculation steps.

Resources:

· Excel Formula for odd numbered questions: “=(A1–1)

· Excel Formula for even numbered questions: “=(5-A1)

* where ‘A1’ equals the cell with the response in it.

· Measuring Usability With The System Usability Scale (SUS)

· How To Use The System Usability Scale (SUS) To Evaluate The Usability Of Your Website

· System Usability Scale (SUS) from Usability.gov

Participants

Another benefit to using the SUS, is you can calculate a valid score from as little as five participants. If you’re able to poll 10 or 20 then so much the better!

What can you do with your scores?

Using an example, let’s say that you have an existing page on your website that needs to be redesigned. You can create an SUS Survey for the existing page, and share it with participants to gain a score. Then once your redesign of that page is ready, give new participants that exact same survey. The goal would be that your score would have increased because the redesign is better than your original webpage.

What you can also do is take the questions within each category (satisfaction, effectiveness, efficiency and learnability) and average the score within each. Then you can see which categories scored the lowest, and focus on improving those aspects on the page.

What’s great about this is you can actually show stakeholders and product managers how your new designs and features are performing and basically justify your existence at a company. Business folks understand metrics and research data, it’s what they use themselves when researching so being able to provide them numbers that coincide with you design projects is essential.

I hope this article was helpful. If you have any questions or critiques, please feel free to contact me! You can find me on twitter or at www.rodixondesigns.com.