Usability testing kit
A collection of guidelines and templates to help you check the usability of election materials.
What you’ll need
This kit covers 5 different approaches to usability testing. Different approaches require slightly different materials, but these are the basics that will get you started:
- Someone to run the test
- Someone to help observe and take note
- Someone who might use what you’re testing
- Something you want to test (a newly designed ballot, a revamped election website, new voting equipment, etc.)
- A quiet place to conduct the test
Getting started
Getting familiar with usability testing1. Getting familiar with usability testing
In a usability test, you show materials to individuals, one at a time. Asking each person to use what you’ve given them, you watch and listen without teaching or training.
That’s different from focus groups, surveys, and advisory groups, where you might show the materials to groups of people and ask them what they think or how they feel.
Why do usability testing
Through usability testing, you can learn about what works (and what doesn’t) for users, and you can confirm or challenge assumptions.
With data from usability testing, you can update your election materials, design a ballot, or launch a new outreach campaign with confidence.
It will help you:
- Identify problems in the design of election materials
- Make it more likely that people will have efficient, effective, and pleasant interactions with your office
- Make it easier for users to use your materials – whether the users are candidates, voters, poll workers, election staff, advocates, media, or other people your office interacts with
When to do usability testing
You can do usability testing almost any time, but it’s most useful:
- Before you start redesigning something
- While you’re still working on a draft
- When a significant event happens that may cause changes or require staff or volunteers to be retrained
- When something about the voting situation has changed since the last election
2. Planning your usability test
What does a usability test look like?
A usability test is made up of one-on-one sessions with one moderator and one user or participant. The moderator (usually you) will be near the participant to conduct the session and ask follow-up questions (but not to teach or help). A note taker may support the moderator by logging observations and discussion.
During the session, the moderator and note taker collect observations (what you see and hear). Observations give you data that you can use to make judgments about what’s frustrating to users and how you might eliminate those frustrations.
Each participant uses a copy of the same materials. The moderator guides participants through the session, observes their behavior, and takes notes (if possible, which is why you have a note taker). Remember: do not help participants do the tasks.
After the testing part is done, if there’s time at the end of the session, you may help participants learn how to use the materials. They often appreciate knowing what their mistakes were. Correcting participants will help you understand what to fix in the materials, too.
Usability test sessions are usually short – about 15 or 20 minutes – but can go longer.
Consider inviting other people on your team to join in by observing the sessions (but silently).
How many participants do I need?
Some rules of thumb:
- Try to test at least 4 people who normally might use the material you’re testing. If you think there might be different behaviors for different types of users, include 4 of each type.
- Make sure your participants represent a diverse group of users. This means including people with disabilities, different ethnic backgrounds, and differing English language proficiencies.
- Don’t worry about demographics like age, education, or income. Think more about what experience participants might have doing the tasks you want to observe, and simply recruit a range.
- You don’t need a large sample of participants. Don’t worry about “statistical significance.”
The goal is to learn enough in your usability test to make informed design decisions.
How should I find participants?
First, you have to decide if you’re going to schedule people in advance, or if you’re going to use impromptu recruiting by going to where likely participants might be.
If you’re scheduling appointments ahead of time, tell everyone. Use every channel you’ve got to invite participants.
If you’re doing quick testing in the field, go to places like community centers, county offices, shopping areas, farmers markets, and libraries. You want to be in a place with a lot of foot traffic.
Looking for some recruitment tips? Check out this article on how to recruit subjects for your usability tests.
No matter what approach you take to recruit your participants, make sure they fit your criteria of likely users. That means that they currently do the behavior you want to observe, such as voting, running a campaign, working the polls, and so on.
What should I look for during a usability test?
You’re looking for both what worked (successes) and what didn’t work (failures). For example:
- Any errors or mistakes – even if the participant corrects them
- Failure to complete or submit
- Places where the participant didn’t follow instructions
- Any signs that the participant doesn’t understand the information or actions needed
- Any errors specific to the material being tested
Behaviors you might see and want to take note of:
- Questions participants ask
- Requests for assistance
- Spoken comments or non-verbal reactions
- Expressions of emotion (confusion, frustration, anger, disgust, delight, satisfaction)
- Any behavior that is adaptive (such as taking out reading glasses or moving in closer to the machine)
- Incidents that would be documented and recorded
- Behaviors or comments that surprise you
- Things that surprise the participants
- Distractions (glare, noise, movement)
3. Holding a practice test
Before testing with real volunteers, consider a practice session with colleagues or friends. This exercise will also help you practice note taking.
Since you might not already have materials you want to test, we’re providing a sample: instructions for how to fold an origami rooster.
Your goal with the test? To measure the effectiveness of the origami instructions provided.
What you’ll need
- Moderator
- 3 participants
- Paper sheets to make an origami
- Test materials (see below)
Test materials
To run this test, you’ll need to download the test materials.
Duration of test
Not more than 15-20 minutes per participant
How to do the test
Find a quiet spot to conduct 3 usability tests — one participant at a time. Share the instruction sheet with them along with the paper supplies to make the origami.
Follow the moderator’s guide provided to introduce the participant to the project and guide them through the process. While they’re testing, jot down observations.
After you’ve completed all 3 sessions of your usability test, use the note taking guide provided to record your top 5 observations and insights about the usability of the origami instructions, focusing specifically on what you heard and what you saw. In the “additional comments” section, include points of analysis on the closing questions:
- What worked well?
- What needs improvement?
Once you’ve practiced with this test session, you’ll be ready to look at the 5 approaches to usability testing that we cover here and think about which one best suits your situation and your needs.
Using the toolUsing the tool
1. Overview: Five approaches to usability testing
There are lots of ways to conduct a usability test, and the approach you take will depend on what you want to learn and the amount of time and resources you’ve got.
We’re highlighting 5 approaches, ranging from quick and easy to thorough and structured:
- Testing in the office: quick usability feedback from colleagues. If you’re in the early stages of designing your materials, this approach is a fast way to get a second opinion.
- Testing out of the office: catch real users in the field. If you have drafts or prototypes that are close to complete, use this approach to test how people use them.
- Testing in the conference room: thorough, structured testing with real users. If you have solid, revised materials, this approach allows you to run a structured test with real users who you’ve scheduled ahead of time.
- Testing by videoconference: thorough, structured testing–remotely. Use this instead of testing in the conference room if you want to include real users from across your jurisdiction or make it easier for them to participate.
- Testing at demo days: collect usability feedback when you are demonstrating new voting systems or forms. If you have materials that you can share in a community review, use this approach to learn about usability during the event.
Below, we provide details, guidelines, and materials for each of the 5 approaches.
2. Testing in the office: Quick usability feedback from colleagues
Use this approach any time you want quick feedback on something you’re developing. Of the 5 approaches, it requires the least time and resources and usually works best in the early stages of designing new materials or updating old ones. It’s also good exercise if you’re new to usability testing.
Who to test with
Bring together 2-4 colleagues – but make sure they’re people who know nothing about the material you’re testing. Although your colleagues need not represent your target audience, you can ask them to use personas to think about a user’s perspective.
If you’d like some tips on how to use personas, check out this article about different ways to approach a form.
What you’ll need
- A moderator
- A rough draft (even a sketch) of the new material you’re developing
- Test materials (see below)
Test materials
To run this test, you’ll need to download the test materials:
Where to do it
Since it’s informal, you can do this test at your desk, a participant’s desk, or in a conference room.
How to do it
- Recruiting – Walk down the hall or to the next department to find someone who’s not familiar with the material. Ask the person if they have 15 minutes to spare to do a quick review of the material with you.
- Setting up – If they agree, find a quiet spot to conduct the test. Take out your moderator’s guide as well as the materials to be tested. Using just a notepad or the note taking guide provided, jot down observations and comments.
- Testing – Hand them the sketch or draft design and ask them to use it as a typical user might. Pay attention as they work their way through the material, jotting down any usability issues that come to the surface during the testing.
- After the test – Thank them for their time. Then, begin the usability test. Pay attention as each participant works their way through the material. Observe their behavior and make note of their comments as they go. When they’re finished, ask them to tell you 2 good things about the materials and 2 things that could be improved. End by recording any demographics or information about their voting experience.
- Reporting – There’s no need to formalize your findings; just use your rough notes on usability issues to revise the materials or start developing a prototype.
3. Testing out of the office: catch real users in the field
Use this approach if you’re looking for quick input on prototypes that people can try out. This test is more structured than testing in the office, but it’s quicker and easier than the testing in the conference room approach.
Who to test with
Your participants should come from the public, so they’re more like your real users (than your colleagues usually are). For example, if you’re testing something for voters, you can ask people if they’re registered to vote and go from there.
Depending on the time you have available for each session, you can work with anywhere between 4 and 8 participants for this test. You don’t need to pay your participants, but you might consider giving a small token of appreciation — like a promotional item provided by your election office.
What you’ll need
- A moderator
- A prototype/early version of the new material you’re developing
- Name badges
- Test materials (see below)
Test materials
To run this test, you’ll need to download the test materials:
Where to do it
The idea for this test is to leave your office and go to a public place. That might be the foyer in your municipal or county building, a public library, or a community center.
If you have some flexibility with location, find a place where you’re likely to find people who are like your real users.
How to do it
- Setting up – Set up a table at the location. Be sure to use signs and badges to tell people what you’re doing (“Help us improve our election materials!”).
- Recruiting – Wearing your badge and a smile, walk up to someone and ask if they could give you a few minutes to help voters in the community. Ideally, you want to find participants who would typically use the materials you’re testing.
- Testing – When they’ve agreed to help you out, have each participant complete the consent form. Then, begin the usability test. Pay attention as each participant works their way through the material. Observe their behavior and make note of their comments as they go. When they’re finished, ask them to tell you 2 good things about the materials and 2 things that could be improved.
- After the test – Review your notes and make sure you haven’t missed anything you wanted to observe or follow up on. Thank your participant for their time and give them a copy of the “thank you” card.
- Reporting – Depending on your situation, you can use your findings to compile a short report or simply use your notes to iterate your design further.
4. Testing in the conference room: thorough, structured testing with real users
Use this approach if you want to conduct a more structured test with participants who are your actual users — not just colleagues or people on the street. A full usability test can happen at any stage in the process, from testing the original version, testing drafts, or testing the almost-final version. This approach may or may not result in a formal report.
Who to test with
For this test, select participants who represent the intended audience for the material being tested. Screen them through phone conversations and schedule their sessions.
You should test with at least 4 participants per user group who will interact with your materials (e.g., 4 regular voters, 4 new voters, etc.). Full usability tests can be conducted with anywhere between 8 and 100 participants. Most of these usability tests include 8-20 participants in individual sessions that last between 30 and 90 minutes.
Depending on your resources, you may choose to compensate participants for their time, especially if each test is a long session of 30 minutes or more.
What you’ll need
- A moderator
- A helper/note taker
- A complete copy of the material to be tested
- Name badges
- Test materials (see below)
In some instances, additional observers may also be included in the testing room. Or, you can use a webcam to have stakeholders observe from another room.
Test materials
To run this test, you’ll need to download the test materials:
Where to do it
Find a quiet place for testing (like a conference room) that will allow you to have substantial, uninterrupted conversations with each participant. If your test includes observers, make sure there’s room for everyone. Alternatively, you can also schedule sessions in a usability lab (two rooms divided by a glass panel to keep participants and observers separate).
How to do it
- Recruiting – Recruit your participants in advance. Schedule each session for between 30 and 45 minutes to give you enough time to conduct the test, jot down notes, and get ready for the next participant. We recommend having at least a half hour break between sessions.
- Setting up – Get to your testing location early to set up. Using the test materials, you’ll have a number of different forms to keep track of. Having them laid out prior to testing will ensure a smooth session. Make sure to use the moderator’s guide and note taking guide provided so that questions and answers can be systematically recorded.
- Testing – First, have each participant complete the consent form. You can have them fill out the demographic questionnaire before or after the test. Following your moderator’s guide, carefully observe how each participant interacts with the testing material as well as any comments/gestures they make during the process.
- After the test – Thank them for their time and give them a copy of the “thank you” card on the way out.
- Reporting – Using the report template provided, compile your results into a report, highlighting all stages of the usability test — i.e., participant data, testing locations, material tested, findings, and recommendations.
5. Testing by videoconference: working with remote users
Use this approach to get feedback from people who can’t physically be in a room with the moderator and note taker. This approach limits participants to people who have technology to access a videoconference, but can be a good way to reach people in a wider geographical area or who might not come to an in-person test.
Who to test with
For this test, select participants who represent the intended audience for the material being tested. Screen them through phone conversations and schedule their sessions. Prior to the test, agree on the videoconferencing platform that can be accessed by all test participants.
You should test with at least four participants per user group who will interact with your materials (e.g., four immigration applicants, four travelers by plane, etc.). Since these users are remote, their sessions will be conducted individually. Full usability tests can be conducted with anywhere between eight and 100 participants. Most of these usability tests include 8-20 participants in individual sessions that last between 30 and 90 minutes. Most will be about an hour long.
Depending on your resources, you may choose to incentivize participation (e.g., stickers or other promotional items), especially if each test is a long session of 30 minutes or more.
What you’ll need
- A moderator
- A helper/note taker
- An email with the material to be tested provided through links or attachments
- Test materials (see below)
- A laptop with a web camera and microphone
- A reliable internet connection
Before the testing, ensure the following is provided to the remote participants:
- Calendar invitation with a link to the digital meeting room
- A reminder to test the link and ensure it is working prior to joining the session
- A reminder to join 10 minutes early to allow for video, audio, and internet connection troubleshooting
If you have additional people observing the test, be sure to go over their role, so they know what to expect. Ask them to keep their video and audio off so they are not distracting. You may also want them to identify themselves with their organization or role, rather than by name.
Test materials
To run this test, you’ll need to download the test materials:
Where to do it
Find a quiet place for testing (like a conference room) that will allow you to have substantial, uninterrupted conversations with each participant. If your test includes observers, make sure there’s room for everyone.
How to do it
- Recruiting – Recruit your participants in advance. Schedule each session for between 30 and 45 minutes to give you enough time to conduct the test, jot down notes, and get ready for the next participant. We recommend having at least a half-hour break between sessions.
- Setting up – Get to your testing location early to set up. Using the test materials, you’ll have a number of different forms to keep track of. Having them laid out prior to testing will ensure a smooth session. Make sure to use the moderator’s guide and note taking guide provided so that questions and answers can be systematically recorded.
- Testing – First, have each participant complete the consent form. You can have them fill out the demographic questionnaire before or after the test. Following your moderator’s guide, carefully observe how each participant interacts with the testing material as well as any comments/gestures they make during the process.
- After the test – Thank them for their time and email them a copy of the “Thank You” card.
- Reporting – Using the report template provided, compile your results into a report, highlighting all stages of the usability test (i.e., participant data, testing locations, material tested, findings, and recommendations).
A note on remote presenting for testing forms – It is best to display the material on your screen using the screen sharing function, pass control to the user, and then allow the user to digitally fill out the information through the screen sharing functionality.
6. Testing at demo days: collecting feedback in a community review
Use this approach if you have an opportunity to get feedback during a public event where you demo elections systems for voter education or as part of a community review. Like the testing out of the office approach, it is another way to collect feedback from people. In this approach, you would be adding some structured usability questions to get more information after they have tried something out for themselves.
This is a way to be sure you learn about the usability of the materials from the people who come to the event. Your focus might be a specific system or you might also include voter information, instructions, or forms.
Who to test with
You will test with anyone who comes to the demo day. This is a good opportunity to invite people from different communities. You might ask local voting advocacy groups to urge “regular voters” to attend, so you get input from all kinds of voters.
What you’ll need
- One or more moderators
- Any systems and copies of other materials you are testing
- Clipboard for each moderator
- Test materials (see below)
Test materials
To run this test, you’ll need to download the test materials. The most important difference in the materials for this kind of test is that we use a form that combines the moderator’s guide and demographics with a simple way to take notes. This is to minimize the paper when working standing up.
One way to keep everything together is to put all the material into a loose-leaf notebook or have it spiral bound at a local copy shop. Copies of the consent form can go at the back. There are two versions of the consent form. One is a basic form for individuals. The other is more like a sign-in sheet if you are keep track of who you worked with and what organizations they represent or what role they play in elections.
Where to do it
You will work at the demonstration location. You will need an area outside of the traffic flow where you can talk to people after they try out the system or other election materials.
- If you are able to help set up the space, you can build this in. Any alcove or a table that people see as they exit will work.
- If you aren’t one of the organizers, get there early to look at the space and find a place to greet people and then talk to them that doesn’t disrupt.
How to do it
- Setting up – Check the space set up for the demonstrations and decide where you will approach people to ask them to talk to you and where you will work with them. Set up any table or chairs. Be sure to use signs and badges to tell people what you are doing. If there is a check-in table, you may want to have someone at that table (or a sign) that lets people know about the testing.
- Recruiting – Wait until people have finished trying the system or other materials. Then, wearing your badge and a smile, walk up to someone and ask if they could give you a few minutes to give you feedback.
- Testing – When they have agreed to participate, walk to the area where you will talk to them. Start by asking where they are from and why they came to the demo day, so you have some context for their feedback. Then begin the usability test. Use a copy or print-out of the election materials to help them explain what worked and what was confusing in any way when they tried the system out on their own, so they can point out details. You should have some specific questions about any aspect of the materials you want more feedback on. When they’re finished, ask them to tell you 2 good things about the materials and 2 things that could be improved.
- After the test – Review your notes and make sure you haven’t missed anything you wanted to observe or follow up on. End by recording any demographics or information about their voting experience. Thank your participant for their time and give them a copy of the “thank you” card.
- Reporting – Depending on your situation, you can use your findings to compile a short report or simply share your notes with the team to use in improving the products.
No matter which approach you take, a usability test will give you important information about how real users interact with your materials, helping you to make informed decisions about how to update or fix issues before you launch the materials out into the world.
You can think of each test as a step toward a more intuitive voting experience!
