Skip to Main Content

A Better Way to Report Accessibility Testing Results?

taught by: Chris Law

Session Summary:

There are currently two primary means for reporting website accessibility testing results: (1) company-specific formatted test results, and (2) VPATs. Even if they’re done well these means present a number of problems for the receiver of the reports that accessibility testers might not be aware of. In this session we detail what those problems are, and present a new, third means of reporting aimed at solving problems for report readers. Prototypes are being developed (and voted on) as part of a new initiative for generating industry consensus standards in accessibility test reporting.

Description:

In this session we present concepts for adding a new, improved third means of accessibility test reporting that’s voted upon as an industry consensus standard.

There are currently two primary means for reporting website accessibility testing results, and they’re both laden with problems for those who need to act on those test results.

  1. The first reporting means is your company’s chosen format for conveying to your client what is wrong, and how it should be fixed. The problem with company-specific reporting tools and methods is that they are usually proprietary, and even if they have been carefully designed and have internal consistency, external consistency can be a big issue. For example, developers in a client organization may use terminology that differs to that of the testing service provider; developers who have been trained using one type of test service provider have to adjust when they change jobs; and people making judgement calls on remediation based on multiple testing sources may be presented with conflicting results that may be hard to compare because of inconsistent formats and terminology.

  2. The second means is more of a sales tool, the Voluntary Product Accessibility Template (VPAT), which the format owners describe as “the leading global reporting format for assisting buyers and sellers”. One of the main problems with the VPAT is that even though the table format is consistent on each report, the content format forces the reader to go on a ‘scavenger hunt’ to find out whether the product is actually accessible. A VPAT might run to thirty or forty pages, and the one piece of information you need is on page 26, but only if you know where to look and what to make of obscure verbiage. There is no statement on page 1 of whether this product passes accessibility tests. For most procurement personnel, who are not used to accessibility terminology, the job of comparing VPATs from different vendors is much more difficult than many people in the accessibility field realize. (In some cases, we have found procurement personnel who mistakenly believe that if the product has a VPAT, that means it is accessible… so they check the accessibility box and move on without even opening the VPAT.)

A number of industry players have got together to start addressing these problems. We are in the process of developing a one-page industry consensus reporting format to supplement the two main existing means of reporting. We are also developing a means of validating test processes to verify that they can create reliable test results. During 2019, the work of developing a base of stakeholders was conducted. In 2020 we are developing the first reporting standard. In this session, we will discuss and present in detail the background needs that are driving this effort, and present the latest prototypes for consistent reporting. We will also describe the work and make-up of the member-organization for forming industry consensus.

Practical Skills:

  • Understand the hidden complexities of generating accessibility test reporting using current means
  • Be able to consider report format issues from the receiver’s point of view (programmers, procurement officials, accessibility program managers, etc.)
  • Understand what types of new reporting formats will be coming in the near future, and how to access and use those new reporting format prototypes.