In my last four posts I shared notes from plenary sessions and workshops I attended at last week’s  2015 Section 508 Interagency Accessibility Forum for Federal Government employees and contractors, including:

This last post includes my notes from a workshop presented by Section 508 Program Managers from the Social Security Administration (SSA) and the Department of State on methods for measuring Section 508 program performance in the context of the OMB reporting framework.  This was a great session and a compelling compliment to DAMM® and other maturity models in the market.

How to Measure Your Agency’s 508 Program

This workshop on measuring the efficacy of a program was presented by Robert Baker, Section 508 Program Manager at Social Security Administration, and Norman Robinson, Section 508 Program Manager at Dept. of State. This was a great presentation – well-structured and well attended (literally standing room only).

  • Talk Overview
    • OMB requires agencies to measure their 508 program and report progress every six months
    • Are you struggling to figure out how to measure compliance? Having difficulty with defining sample sizes and rolling up results? Not sure you understand what is expected?
    • The focus for this talk was about how to test a 508 program to see if it is mature and effective
    • Both SSA and DHS, Norman’s old employer, have a history of measuring and testing programs for efficacy
    • Combination of a 508 and 504 team under one office held in HR
  • What is a 508 program?
    • Do you actually have a program? Are you one of the coordinators or the coordinator?
    • Do you have policy? Governance? Budgeted resources?
    • There are people who have been a coordinator for many years, but do not have a program because they don’t have any budgeted resources.
    • OMB requires agencies to measure their 508 programs and report on program efficacy every 6 months
  • Your Questions
    • What if you have lots of data points but don’t have a clear way to quantify how a big program is compliant?
    • What are the specifics of the OMB compliance requirements?
    • How do we deal with enforcement? We have this requirement but we consistently don’t comply.
    • What are the processes and tools for aligning the program?
    • How do you make your program align with the strategic initiatives?
  • Lies, Damn Lies and Statistics, and Representative Samples
    • You can’t manage what you can’t measure…but you can make the numbers say pretty much what you want
  • OMB Template
    • The OMB Template contains a narrative which is where you can tell your story. The narrative should explain the approach and help in explaining the metrics reported. The core idea is that you need to use this to explain how your program uses metrics to increase its efficacy over time.
    • OMB defined a common framework for metrics but they didn’t define a specific method for implementing the program. That’s the point – you have to report on the impact and approach but not on the exact method for implementing things.
  • OMB Reporting
    • CFO Act requires that you report to OMB on your 508 compliance program on a rolling six month basis
    • Currently all the agencies that are required to report are reporting. The departments may not be aware of it but the agency is reporting.
  • Maturity Definitions
    • The report requires reporting through for level of maturity in each of the areas
    • Four levels are – Adhoc, planned, resourced, measured
    • These are less formal levels than in a CMMI model and framework.
    • Maturity measures = process
    • Compliance measures = products (things)
  • Measures Definitions
    • 508 Compliance Validation Results (% meeting technical standards of those evaluated / tested)
    • Maturity Levels: Procurement Solicitations, EIT Life Cycle Activities, Testing, Training, Complaints
    • Don’t have time to go through all the agency items, only a small sample, so what does that mean for reporting?
  • What is the purpose of the OMB Template?
    • Enables OMB to set expectations, gauge status and monitor progress
    • Enables CIO Council Accessibility Community of Practice to (i) assess need for support and (ii) prioritize best practice and training efforts
    • This provides the data for the CIO Council to guide and drive government wide areas of need where impact can be made across the U.S. Federal Government
    • A key focus is coordination across the Federal Government and having good data to make policy decision on
    • The OMB approach is not about a witch hunt – its a tool to mature both specific programs and the federal approach as a whole
  • How Metrics Benefit Your Agency
    • Measuring your 508 program enables you to identify and manage risks, gain management support, mature your 508 program in both breadth and depth
    • SSA story:
      • Inserted gateways into the development lifecycle, added in testing into all projects and then started collecting data
      • As data was collected it was reported back out to the individual business units
      • Without setting a goal for compliance, just by exposing the data, things started to change
      • The second step was to show the level of compliance of applications relative to their peer applications throughout the agency – this starts to create a competitive environment
      • By not giving management a target, just providing them data and support, within four years we went from 72% to 94% and have sustained that for four years
      • Effective measurement information provides one of the greatest tools to engage with your agencies
  • How to Get Started
    • It’s not about enforcement it’s about what you can do and how to start making progress
    • OMB isn’t concerned about black or white conformance – the focus is starting somewhere and starting to make some progress
    • To get started don’t try to test everything to create a baseline – test a small sample set and then increase the scope of that over time
    • Start small, find a way to measure it, and show the data to someone that can impact that component
    • “We don’t get a lot of progress in 508 because someone is bleeding. We get a lot of progress because people are confronted with hard data that they are failing” -RB
    • “I cannot stress to you enough how important it is to simply show a dashboard that is red. It shows in a clear, documented fashion that something is not happening in the way that it should” -NR
    • OMB is meeting with the CIO on a quarterly basis as part of portfolio status
    • On a quarterly basis the management is getting questions and the CIO needs to have substantive answers for those issues
  • How to Count 508 Program Resources
    • Resources allocated to your 508 program support include annual IT dollars spent on people who conduct 508 program activities – full or part time, FTE and contractors, and purchase of 508 tools
    • This is important since it clearly exposes the need for resources to put in place an effective program
  • How to Define Scope
    • Where is the most work being performed?
    • Are you covering the areas where the most significant work is being performed?
    • Use the narrative to discuss progress and plans to extend practices across components
    • What can you measure?
    • Is that representative of the agency?
    • Measure that which can be measured, set a goal to expand that based on the greatest areas of risk
    • Narrative can also capture the scope of the sample as an aspect of how this all works
  • How to Define Maturity Levels
    • Planned
      • Do you have a framework to define your agency actions?
      • Is that framework defined at the specific or general level? Is it actionable?
      • Does that framework cover the areas where the most significant work is performed?
      • Are you mostly there? Give yourself credit.
    • Resourced
      • Have people been trained on how to do the policy?
      • Have they been provided guidelines and training to succeed?
      • Are they actually doing the work? Are there specific artifacts that are actually being produced on a day-to-day operational level?
      • Are you mostly there? Give yourself credit.
    • Material Weaknesses
      • If resources are not dedicated in a key area – call it out
      • If measurements are not happening against key systems – call it out
    • Measured
      • Do you have an ongoing process to validate the resources are following the policy and procedures?
      • Do you use the measurements to make changes to your 508 program
  • How to Calculate a Conformance Metric
    • Do you have a test process? If you don’t have a valid test process all the testing data won’t be valid
    • How do you calculate a representative sample?
    • Should you test if you are not planned or resourced to test?
    • What are you resourced to measure?
    • Can you make data driven decisions from your sample?
    • What can you work towards (depth or breadth)?
      • As an example SSA.gov has 25K pages. You can buy a tool to test all pages and cover about 25% of the requirements or you can get a testing team to test 25 of the pages fully. Either approach can be compelling it just depends on how the asset is setup.
  • How to Calculate Conformance
    • Percentage of applicable 508 standards supported
    • Pass / Fail for each standard
    • The former approach is a more subtle and accurate approach but needs to have more detailed data for scoring
  • How to Aggregate Results
    • Establish expectations for how conformance will be measured on the aggregated up front
    • Non-weight average – each item gets the same weight
    • Weighted average – Each items gets a weight
    • Multiweight average – Multiple factors are weighted in the roll up
  • What measurement can do for you?
    • Drive enterprise application 508 compliance
    • Raise awareness among CIOs and executives about risks they don’t know
    • Provide a communication tool to focus attention on program needs
  • CIOC 508 Reporting Template [pdf]
  • 508 Reporting Template Instruction Manual [docx]

You can access the slides from this presentation at www.peatworks.org/federalforum.

Published 5 years ago