Automated Accessibility Testing is Not a Silver Bullet

We live in an era of rapid technological advancement, where automation has become synonymous with efficiency and productivity. You’ve probably seen those YouTube ads hyping up AI as a job-saving superpower or pushing expensive masterclasses. From self-driving cars to automated customer service, it seems like there’s a machine ready to replace human intervention at every turn. In the realm of software development, automation has undeniably revolutionized testing processes, including accessibility testing. 

As a UX/UI designer focusing on accessibility at DigitalA11Y, I’ve seen firsthand how both automation and manual testing play crucial roles during accessibility audits. 

However, while automated tools are invaluable for catching those small number of accessibility errors and ensuring basic compliance with accessibility standards, they cannot replace the human touch when it comes to performing manual accessibility audits. 

The Limitations of Automation

Automated accessibility testing is undoubtedly a powerful tool. It can quickly scan through code, identify missing alt text, incorrect color contrast ratios, and other common accessibility issues. This allows developers to focus on more complex tasks and ensures a baseline level of accessibility. 

But let’s be clear: automation is not a silver bullet. It cannot replicate the nuanced understanding of human perception, cognition, and behavior.

According to a study conducted by Equal Entry, automated tools are able to identify only a limited number of accessibility issues. The data indicates that only a small portion of accessibility failures are captured by these tools.

Here are some of the limitations of automated testing:

• Subjectivity and Context: Automated tools cannot evaluate subjective elements like clarity, understandability, or cultural appropriateness. They also lack the ability to comprehend content within its broader context. 
• Complex Interactions: Automated tools struggle with complex user interactions, such as navigating through dynamic content or using assistive technologies. 
• False Positives and Negatives: Automation can sometimes produce inaccurate results, leading to wasted time and effort. 
• Limited Scope: Many WCAG guidelines require human judgment and cannot be fully assessed through automation. 

Empathy: The Heart of Accessibility

Accessibility is fundamentally about empathy. It’s about putting yourself in the shoes of users with disabilities and understanding their challenges. Automated tools can identify technical barriers, but they cannot instill empathy.

Accessibility is empathy-driven, and you can not automate empathy.

I am not an accessibility tester, but as a background in User experience, I understand empathy is so important. And that’s why at DigitalA11Y :

The first thing we do is try using the website with just a keyboard. We see if it’s easy to move around. We check things like pop-ups, how to close them, the main menu, and how everything works together. We think about if this would be easy for someone who is visually impaired or low vision. 

Next, we use a screen reader. At DigitalA11Y, we have team members who are screen reader users so they know what to look for and provide valuable insights. 

Explore our blog for a guide on Performing an Accessibility Audit,

There are many ways to check for accessibility problems, but here are three reasons why people are better than computers: 

• Comprehensive coverage: Manual accessibility testing can evaluate all WCAG criteria, providing a more accurate assessment of accessibility of product or service. 
• Reliability: Human testers can provide more reliable and trustworthy results.
• Tailored recommendations: Accessibility experts can offer specific solutions based on their findings.

To create truly accessible digital experiences, we need to go beyond technical compliance. We need to:

• Understand the Diverse Needs of Users: Conduct user research to learn about the specific challenges faced by people with different disabilities.
• Test with Real Users: Involve users with disabilities in the testing process to get their feedback on the usability and accessibility of your product.
• Foster a Culture of Inclusion: Create a workplace culture that values diversity and inclusion, and prioritize accessibility in all aspects of product development.

A Balanced Approach

While I believe that manual accessibility testing is crucial to ensure that your product meets the needs of all users, automated tools like AXE and WAVE can be helpful for initial checks. They are easy, fast, and affordable. However, they cannot catch every issue on their own.

The key to effective accessibility testing is a combination of automation and manual testing.

Incorporating both automated and manual accessibility testing ensures a more thorough and effective approach to accessibility, benefiting all users. You can:

• Identify a Wider Range of Issues: Catch both technical and usability-related accessibility problems.
• Prioritize Fixes: Focus on the most critical issues that impact real users.
• Continuously Improve: Make accessibility an ongoing process, rather than a one-time checklist.

Conclusion

In conclusion, while automation is a valuable tool in the accessibility toolkit, it cannot replace human judgment and empathy. To create truly inclusive digital experiences, we must prioritize Manual Accessibility Testing and user-centered design. By working together, we can build a more accessible web for everyone.