Usability Study for Accessibility
We tested a pharmaceutical giant’s career website for accessibility issues, running a study with participants who had a disability in one or more of three areas: visual, motor and/or cognitive. We conducted the study in our usability lab, planning, recruiting, executing and presenting back to the client in just over 4 weeks.
GOALS
- Evaluate the usability of the 'Careers' section of the client's website for employment candidates with disabilities.
- Develop a set of specific design recommendations which will improve the usability of key tasks in the career section for candidates with disabilities.
APPROACH
Usability Sessions: Nine one-on-one, 60-minute sessions were conducted over 3 days. One of these people person participated remotely via GoToMeeting, and the others came to our usability lab. Half of the participants were recruited by a recruiting agency, and the other half were recruited through a partnership with the Perkins School for the Blind in Watertown MA. Participants received $100 compensation for their time.
Expert Review & Moderator's Guide: To inform the moderator's guide that was used for the usability sessions, I created an expert review for internal use that outlined all potential usability issues (large and small). I presented this review to my team to give us all the same understanding of the site and how we might structure the usability sessions. This particular website was a bit disjointed, with the "Careers" section outsourced to a third party, and two different sub-sites within it. The review helped identify the main sections of the site we should target (we couldn't get everything in 60 minutes) and what portions to watch out for or take participants back to if they were bypassed. I wrote the moderator's guide to take participants through three main tasks: 1) Exploring the Careers website; 2) Finding/Searching for a job; 3) Applying for a job.
To uncover all potential accessibility problems, we recruited Participants with a variety of disabilities who used a wide range of accessibility tools.
The moderator's guide was written to lead participants through three major Tasks: Exploring the "careers" website, Searching for a job, and applying for a job.
ANALYSIS
We took session notes in a word document, using timestamps in the recording wherever possible. Because we had only 9 people in the study, all with different disabilities, reporting issues was a big challenge. In many cases, only one or two people encountered a specific issue, but when they did, it caused them a lot of trouble. We used the outline of the moderator's guide and the flow of the session to report usability issues found during each key task, which coincided with a general area of the website. After overall usability issues were reported, we included slides devoted to the particular difficult spots for people with Visual Impairments, people with Motor/dexterity issues, and people with cognitive impairments. This allowed us to feature broad, overarching usability issues as well as more isolated or one-off trouble spots. It was hard for us to prescribe solutions for all of the more isolated or one-off issues, and many accessibility-specific recommendations began with "Consider..." (e.g., "Consider increasing the contrast between header 2 text and the background").
In addition to the challenge of creating an all encompassing report that didn't lead the client astray by focusing to much on the details, the difficulty reporting was compounded by the fact that there was an additional team between us and the client. Despite not being usability experts or attending any of the sessions, this team had strong opinions on how the report should be structured and worded - ones that we did not always agree with. In the end, we made several iterations of the report and came to an agreement about its flow, look, and feel. But, it was massive... 88 cluttered slides! This is something that I wish we could have cut down on a bit. However, for one-time work like this study, the report is the only chance we have to get across everything the client needs to know, and we couldn't afford to leave anything out.
Key Takeaways when designing for Accessibility:
- Usability issues affect those with disabilities much more than the average user.
- Especially good error-handling is needed
- Labor-intensive recovery from mistakes and general data entry
- Importance of alt-text & descriptions
- Participants with visual and motor / dexterity issues had trouble accessing content that was activated upon hover (menus, search, and interactive graphics).
OUTCOME
The report was received well by the client and their team. However, I believe that they could have uncovered many of these issues in an expert review conducted by a usability or development expert who is well-versed in the Web Content Accessibility Guidelines (WCAG 2.0), rather than having participants be the ones to stumble over easy to find problems first. Looking back, it would have been much more effective for the client had they fixed the website as best they could, omitting low hanging fruit (e.g. lack of descriptions and alt text) and then tested the site with participants after that to truly validate the site. It wasn't quite the right method for them at that particular point in time. In our study, participants were eager to share their troubles, but when they were slowed down by usability issues, they were exponentially slowed down... in many cases we were not able to get through all tasks in the time allotted.
From a practitioner perspective, there were many things we learned after running this test. We wanted to share them and wrote a blog post titled "Usability Studies and Participants with Disabilities: What You Need to Know."