Most people today will agree that testing for accessibility is a good idea. Unfortunately, it is unclear how best to test for accessibility in a fashion that provides a reasonable degree of testing coverage for compliance requirements.
In general, most tests for accessibility in place today utilize an automated testing tool to find violations. These approaches concern themselves with testing a given IT system, usually a web site, against a set of automatic tests. This approach has some significant benefits: it is generally cheap, easy to use, and fast. While cheap and fast, however, it is effective only at validating a certain sub-set of accessible requirements and cannot provide a comprehensive validation of accessibility. The scope of coverage varies amongst tools but is generally around 30% of the total accessibility requirements.
As an example, nearly all tools can test for the presence of alternative text, but no tools can test if the alternative text is a meaningful replacement for an image. As a second example, tools can test for a form field label, but not whether an assistive technology user can fill out a form – the requirement of most accessibility standards.
Automated tools tend to be limited to a specific technology platform. Tools that test Web content are not going to test your PDF content. Additionally, tools do not exist for many major technology platforms in wide use today.
For the purposes of evaluating Section 508 compliance, in particular, it should be noted that the requirements of Section 508 functional standards (§1194.31) by definition must be evaluated with Assistive Technology (AT) testing. As such, testing for Section 508 compliance requires testing both normative – or rules-based – and AT based testing. Thus without the completion of AT testing, you will not have information to create a responsible claim of compliance with the Section 508 requirements.
In sum, automatic testing is a cheap and easy way to validate a sub-set of compliance violations quickly. In general, however robust or well-developed they are, these tools all suffer from the same limitations on what can be tested automatically. Thus, if automated testing is all your organization uses to determine compliance, you will be unable to make a responsible claim of accessibility or compliance.
It is important to educate anyone on the importance of the human aspect in accessibility testing (beyond just automated tools) and on the need for functional testing. Additionally, the people involved in user testing (functional and manual) must have a deep knowledge of accessibility issues and solutions. This includes the applicable laws and standards they are evaluating against, the automated testing tool that they are using, the barriers that people with disabilities experience, and the assistive technologies that users with disabilities use on a daily basis to gain access to digital systems.
Lastly, it is worth noting that when we consider disabilities, we are not just considering blindness, but the entire spectrum of disabilities, including ones that we cannot identify visually. These considerations can also help those with temporary disabilities or certain contextual needs in addition to other, more general benefits.