Agile Accessibility KPIs: How to Measure What Matters

If your team has ever worked overtime to address every finding in an accessibility audit, only for the same issues to re-emerge before remediation is anywhere near complete, you’ve been stuck in a break/fix cycle. Thankfully, there’s a way to escape (or avoid) this conundrum—and if you’re caught up on our recent content, you’re likely already familiar with it. We’re talking, of course, about agile accessibility.

In an agile approach, teams tackle remediation in focused, high-priority segments, and proactively embed accessibility into digital experience creation to avoid introducing new accessibility barriers. It’s a formula for efficient fixes and sustainable momentum—but there’s one challenge.

While it can be fairly straightforward (if onerous) to track progress toward resolving a large-scale audit’s findings, when accessibility is seamlessly embedded into your team’s everyday work, it can be tricky to understand how to track your impact. Do you measure the number of overall accessibility issues in your digital properties? Or how many critical barriers exist? Or the time it takes to address accessibility issues? All three?

This blog will offer three key performance indicators (KPIs) that product teams can use to monitor the effectiveness of their agile accessibility efforts, as well as two process-oriented accessibility KPIs to help in shifting mindset and focus if your team is in the process of adopting a more agile approach. Seem like a lot to keep track of? We’ll end off with an explanation of how our solution can enable you to bring together data in an agile way and stay focused on what matters most.

Why measure the success of your agile accessibility efforts?

If you haven’t really started tracking accessibility progress for each of your digital experiences, you’re not alone. Many teams don’t have any ongoing KPIs for digital accessibility, either because they’re unaware of their obligations or because they treat accessibility as a one-time, box-checking activity, where the completion of remediation on an audit’s findings is equated with a team having “solved” accessibility on their site.

This reality is unfortunate because consistent measurement is crucial to building sustainable, accessible, usable digital experiences for everyone, and supporting your broader organizational inclusion and compliance goals. Teams need current data to understand where their efforts are having an impact, what blockers they need to get past, and how they may need to work differently to achieve their desired results—and that counts double for product and development teams working in agile development methodologies, who need data to be able to pivot and iterate on plans in real time.

Plus, accessibility measurement doesn’t just help teams work smarter—accessibility KPIs are also important for helping other stakeholders within your organization understand the impact of your accessibility efforts, which can help you secure additional buy-in and resources.

What should you measure to understand if your agile approach to accessibility is working?

The accessibility KPIs that your team chooses to track may vary based on your team’s unique objectives. We’ve mapped each of the following KPIs to a common goal of teams employing agile accessibility, so you can choose the metrics that best support what you’re currently hoping to accomplish.

Goal: Decrease bugs that stop users from accomplishing tasks

KPI: Number of critical issues currently live in key user flows

Agile is a user-centric philosophy, and the most common methodologies, such as agile scrum, are built to solve user needs and problems. That’s why the requirements for work to be performed during agile development are written from the perspective of the user. In an agile approach to digital accessibility, teams shift from a “compliance” focus—trying (and typically failing) to eliminate every bug on every page of a digital experience, in no strategic order—toward a usability focus: quickly delivering experiences that allow individuals with disabilities to easily accomplish their goals.

A user flow refers to a specific path that users take to complete core tasks on a digital property, such as making a purchase, booking an appointment, requesting a demo, or accessing important information. Measuring the number of critical issues (issues with the most severe user impact) that are live in key user flows across your site or experience helps your team understand whether your accessibility efforts are being put in the right place and ensures that users’ most urgent access needs are being met first.

Get Started with Agile Accessibility

Download guide

Goal: Decrease the time it takes to resolve identified accessibility issues

KPI: Average amount of time that an accessibility issue sits in the in backlog

Agile is about measuring velocity: how much you can produce in a specific time period at the quality standards your team deems acceptable. But many agile teams don’t include accessibility in those quality standards—either because they don’t understand the severity of various accessibility issues or worry that proactively addressing accessibility will slow down innovation. As a result, every new release contributes to a growing backlog of accessibility bugs.

In agile accessibility, teams build accessibility into their workflows so fewer and fewer bugs slip into live digital experiences. Simultaneously, they become more efficient at remediating bugs that are already live, eventually reducing critical accessibility issues in the backlog to zero.

A reduction in the average amount of time that an accessibility issue remains in the backlog shows that a team is not only maintaining the accessibility of their product, but also improving over time, whether that’s with every few sprints, every release, or along a different cadence they decide to set.

Goal: Reduce new accessibility issues

KPI: Number of accessibility issues introduced in new releases compared to previous releases

Agile methodologies exist to support sustainable ways of work. In fact, one of the 12 agile principles states, “Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.”

When it comes to accessibility, that means building experiences that are accessible by default. By embedding digital accessibility expertise and testing into every phase of the software development life cycle, teams adopting agile accessibility should notice a downward trend over time in the number of accessibility issues introduced in each new release. With this approach, more and more accessibility bugs will be caught before features go live—or even better, before code is even merged. Including accessibility in your Definition of Done (DoD) is an impactful way to avoid creating a new backlog of accessibility issues.

For continued success, don’t just measure the “what”

Of course, for strong collaboration and stakeholder engagement, it’s important to have data on the “what” of digital accessibility: the number of issues impacting users with disabilities in your digital experience, with context on their severity and impact. But if you shift to also focus on the “how,” you can understand what you need to improve your overall digital accessibility approach, such as additional resources or adjustments to velocity (like taking a fix from two to four story points). This information may be less relevant for reporting to supervising stakeholders and leadership, but it’s of crucial importance to agile development teams, as it feeds into sprint planning.

Some important agile accessibility KPIs that indicate how your team is doing with its adoption of agile accessibility might be:

• The number of accessibility bugs caught in the early stages of development, versus after release. This demonstrates that for a development team, accessibility is not an afterthought, and more bugs are getting caught earlier. Because it’s often much faster and cheaper to fix an issue early in the development cycle, this also impacts the overall efficiency of a development function. Notably, for product managers and team leaders just getting started with agile accessibility, the stage at which accessibility bugs are being remediated may not always be immediately clear or easy to surface. If that’s the case for your team, getting clearer insight on “when” these bugs are being caught will be an important first step toward measuring process efficiency.
• The percentage of new designs that include accessibility annotation. This shows that cross-functional communication between design and development, which is crucial to success in agile accessibility, is a priority. If these annotations are being applied (and incorporated) consistently, it’s a positive indication that the larger team or department is embedding accessibility best practices beyond—and before—development.

Agile Accessibility in Product Management guide

Download guide

Too many numbers? We can help.

As we’ve established, tracking accessibility KPIs—or rather, tracking the right accessibility KPIs—is essential for sustainable success. But consistently keeping tabs on multiple data points can be challenging, and you don’t want to spend more time tracking progress than making progress.

The Level Access Platform unifies and organizes findings from both automated scans and manual evaluations of digital experiences, equipping teams with a complete understanding of their accessibility performance in a simplified manner that makes it easier to understand and prioritize issues. With sophisticated monitoring tools, teams can run recurring scans to track the progress being made over time on digital experiences (or individual pages) in different stages, from development, to staging, to production. This allows a more detailed understanding of how accessibility is progressing as an experience takes shape. And, teams can drill down even further by monitoring issues with key design components so remediation efforts can be focused on core building blocks that impact an entire digital experience.

Our digital property-wide, page-level, component-level, and “by-rule” dashboards also perfectly support an agile approach to monitoring by highlighting trends in key issues over time. These flexible views help leaders and managers gain insight on the types of issues that teams are prioritizing for remediation, and the types of issues teams may need training to avoid introducing in the next feature release.

In addition, our unique governance capabilities help teams to set policies for digital accessibility to align around shared goals. For example, if velocity in remediation is a strong priority, teams can set a policy for how long a critical finding is allowed to stay open, and get customized alerts when findings fall outside that policy. Dashboards at both the workspace level (which may involve multiple different digital properties) and organizational level also help reveal common findings, highlighting what areas your teams and organization may need to focus on when it comes to digital accessibility education and training.

Seize the data

When you’re ready to commit to sustainable, lasting progress toward accessibility, we’re ready to help. With a streamlined user interface and powerful monitoring and reporting capabilities, the Level Access Platform is your single system-of-record for tracking and organizing accessibility KPIs as part of your agile approach. Request a demo today to learn how our solution can help you meet your accessibility and compliance goals faster, for good.

eSSENTIAL Accessibility has changed its name to Level Access! Read More