Setting Up Progressive Accessibility Thresholds in CI

The smallest useful accessibility gate is four integers in a file. Scan the application, count violations by impact level, write {"critical":4,"serious":96,"moderate":212,"minor":57} to a committed JSON file, and fail the build when any of those four numbers goes up. This guide is part of Progressive Threshold Management, and it is deliberately the day-one version: no fingerprints, no per-rule keys, no scheduled automation — just the gate that stops the count from growing while the team decides what the real policy should be.

Root Cause

A zero-violation gate on an application that already has 369 violations fails on the first pull request, and every pull request after it, for reasons the author did not cause and cannot fix. The predictable outcome is that the job gets marked continue-on-error: true within a week and then ignored for a year. The gate is not wrong about the application; it is wrong about what a pull-request check is for, which is to attribute a change to the change that caused it.

Counting by impact level rather than by rule or by node is what makes this version cheap enough to ship in an afternoon. axe-core already assigns every result an impact of critical, serious, moderate or minor, so the tally needs no DOM knowledge, no selector handling and no hashing — the only prerequisite is that the run options are fixed, which is the tag set and exclusion list settled once in axe-core configuration and setup. Four numbers also happen to be the granularity a manager can read: “critical is at 4 and has not moved in three weeks” is a sentence, whereas a per-rule table with 63 rows is a spreadsheet.

The cost is precision, and it is worth stating up front rather than discovering it in month three. Because the gate compares totals per level, a pull request that fixes one serious finding and introduces a different serious finding passes. That trade is invisible here, and closing it needs the fingerprinted baseline described in the parent guide. What this version does guarantee is that the totals are monotonically non-increasing, which is the property that turns 369 into a burn-down rather than a running balance.

Record mode and enforce mode On the left, a scan feeds a tally of four impact counts. In record mode the tally is written to the committed ceilings file. In enforce mode the tally is compared level by level with the committed file, passing when every level is equal or lower and failing when any level is higher. axe scan 38 routes tally by impact 4 / 96 / 212 / 57 369 total --record run once, by hand --enforce every pull request ceilings.json committed, reviewed all levels ≤ ceiling exit 0 any level higher → exit 1
One script, one flag: the recorded file is written by hand exactly once and read on every pull request thereafter.

Configuration

Three files: a route list, the recorded ceilings, and the script that does both jobs. Start with the routes, because the tally is only comparable between runs if the set of scanned pages is fixed. A crawler that discovers 38 pages today and 41 tomorrow reports three pages’ worth of violations as a regression.

{
  "baseUrl": "http://127.0.0.1:4173",
  "tags": ["wcag2a", "wcag2aa", "wcag21aa", "wcag22aa"],
  "routes": [
    "/",
    "/search?q=chair",
    "/product/tt-4410",
    "/basket",
    "/checkout/delivery",
    "/account/orders",
    "/help/returns"
  ]
}

The recorded ceilings file is written by the script, but commit it with a comment field so the next person to read it knows when and from what it came. JSON has no comments, so use a real key.

{
  "recordedAt": "2026-07-25T09:14:02.881Z",
  "recordedFrom": "main@8f21c0d, 38 routes, axe-core 4.10.2",
  "note": "Day-one ceilings. Lower these by hand when a level drops for a full week.",
  "ceilings": {
    "critical": 4,
    "serious": 96,
    "moderate": 212,
    "minor": 57
  }
}

Four keys is the whole schema, and resisting the urge to add more is part of the design. No decay rate, no severity weights, no per-branch overrides: a weighted score cannot be reproduced by hand, and a branch override means the number a developer sees locally is not the number the gate applies. If different parts of the repository genuinely need different bars — a new package held at zero while a legacy area keeps its allowance — that is a different mechanism, keyed by path rather than by branch, and it belongs in per-directory accessibility budgets in legacy code rather than bolted onto four integers.

Now the script. It runs the scan, tallies by impact, and then either writes the file or compares against it. Keeping both modes in one file guarantees the tally logic is identical in recording and enforcement — the most common cause of a gate that fails on its own baseline is a recorder that counted violations and an enforcer that counted nodes.

#!/usr/bin/env node
// a11y/impact-gate.mjs
// usage: node a11y/impact-gate.mjs --record | --enforce [--warn-only]
import { readFileSync, writeFileSync } from 'node:fs';
import { chromium } from 'playwright';
import AxeBuilder from '@axe-core/playwright';

const LEVELS = ['critical', 'serious', 'moderate', 'minor'];
const CEILINGS_FILE = 'a11y/ceilings.json';
const mode = process.argv.includes('--record') ? 'record' : 'enforce';
const warnOnly = process.argv.includes('--warn-only');
const config = JSON.parse(readFileSync('a11y/routes.json', 'utf8'));

async function tally() {
  const counts = Object.fromEntries(LEVELS.map((l) => [l, 0]));
  const browser = await chromium.launch();
  // Pin everything that could change a contrast reading between two runs.
  const context = await browser.newContext({
    viewport: { width: 1280, height: 900 },
    colorScheme: 'light',
    reducedMotion: 'reduce',
  });
  for (const route of config.routes) {
    const page = await context.newPage();
    await page.goto(config.baseUrl + route, { waitUntil: 'load' });
    await page.getByRole('main').waitFor({ timeout: 15_000 });
    const results = await new AxeBuilder({ page }).withTags(config.tags).analyze();
    for (const violation of results.violations) {
      // Count NODES, not violations: one rule with nine bad nodes is nine findings.
      for (const node of violation.nodes) {
        counts[node.impact ?? violation.impact ?? 'minor'] += 1;
      }
    }
    await page.close();
  }
  await browser.close();
  return counts;
}

const counts = await tally();
const total = LEVELS.reduce((sum, l) => sum + counts[l], 0);

if (mode === 'record') {
  writeFileSync(CEILINGS_FILE, JSON.stringify({
    recordedAt: new Date().toISOString(),
    recordedFrom: `${process.env.GITHUB_SHA ?? 'local'}, ` +
      `${config.routes.length} routes`,
    note: 'Day-one ceilings. Lower by hand when a level drops for a full week.',
    ceilings: counts,
  }, null, 2) + '\n');
  console.log(`Recorded ${total} findings: ${JSON.stringify(counts)}`);
  process.exit(0);
}

const { ceilings } = JSON.parse(readFileSync(CEILINGS_FILE, 'utf8'));
const risen = LEVELS.filter((l) => counts[l] > ceilings[l]);
const fallen = LEVELS.filter((l) => counts[l] < ceilings[l]);

console.log('| impact | ceiling | this run | delta |');
console.log('|---|---|---|---|');
for (const l of LEVELS) {
  const delta = counts[l] - ceilings[l];
  console.log(`| ${l} | ${ceilings[l]} | ${counts[l]} | ${delta > 0 ? '+' : ''}${delta} |`);
}
for (const l of fallen) {
  console.log(`GOOD: ${l} is ${ceilings[l] - counts[l]} below its ceiling.`);
}
for (const l of risen) {
  console.error(`REGRESSION: ${l} rose from ${ceilings[l]} to ${counts[l]}.`);
}
// --warn-only reports but never blocks; used for the first few days of rollout.
process.exit(risen.length > 0 && !warnOnly ? 1 : 0);

Two decisions in that script are the ones worth defending in review. It counts nodes rather than violation objects, because axe groups every failing node for one rule into a single violations entry — a page that goes from one unlabelled input to nine would otherwise show no change at all. And it prints a markdown table on stdout, which means the step output can be appended straight to a job summary with no extra formatting code.

One enforcement run compared against the recorded ceilings Four rows, one per impact level. Each row shows the recorded ceiling as a light bar and the current run as a solid bar. Critical is three of four, serious is ninety-one of ninety-six, moderate is exactly two hundred and twelve, and minor is sixty against a ceiling of fifty-seven, which is the only failing row. Enforcement run on pull request 1284 light bar = recorded ceiling, solid bar = this run, bar length is scaled per row critical 3 of 4 · pass serious 91 of 96 · pass moderate 212 of 212 · at ceiling minor 60 of 57 · FAIL ceiling REGRESSION: minor rose from 57 to 60 — exit 1, even though the total fell from 369 to 366.
Each level is judged on its own, which is why a run whose grand total fell by three still fails: the gate is four independent comparisons, not one.

Validation

Prove the gate works before trusting it, in three commands and about four minutes. Record, verify the recorded file passes against itself, then plant a violation and confirm the job fails on exactly the level expected.

# 1. Record from a clean checkout of the default branch.
npm run build && npx serve -s dist -l 4173 &
node a11y/impact-gate.mjs --record
git add a11y/ceilings.json && git commit -m "chore(a11y): record impact ceilings"

# 2. Enforce with no changes: must exit 0 with every delta at zero.
node a11y/impact-gate.mjs --enforce; echo "exit=$?"   # expect exit=0

# 3. Plant a known critical failure and confirm attribution.
cat >> src/pages/Basket.tsx <<'EOF'
// temporary: unlabelled control to validate the gate
export const GateProbe = () => <input type="text" />;
EOF
npm run build && node a11y/impact-gate.mjs --enforce; echo "exit=$?"  # expect exit=1
git checkout -- src/pages/Basket.tsx

The third command should print a table whose critical row reads | critical | 4 | 5 | +1 | followed by REGRESSION: critical rose from 4 to 5. and exit 1. If it exits 1 with a rise on a level other than the one the planted defect belongs to, the tally is picking up something unstable — check that the probe rendered and that no other route changed between runs.

Run step 2 three times in a row before enabling the gate as a required check. Three identical tallies is the evidence that the numbers are deterministic; two out of three means a wait is missing and the gate will fail somebody’s unrelated pull request within a day.

One more check is worth the two minutes it costs: re-record on a second machine, or in the container the pipeline uses, and diff the two files. A difference of one or two in serious between a laptop and a runner is almost always font fallback changing a computed contrast ratio at the margin, and it is far cheaper to discover that now than to argue about it on somebody’s pull request. Whichever environment the pipeline uses is the environment the ceilings must be recorded in, and the recordedFrom field is where that decision gets written down.

Edge Cases and Conditional Guards

  • A node with no impact. axe usually sets impact on each node, but a custom rule with no impact in its check metadata leaves it undefined, and counts[undefined] silently becomes NaN in a naive tally. The script falls back to the violation-level impact and then to minor, so an unclassified finding is still counted rather than quietly dropped.
  • A route that renders nothing on a runner. If a page 404s or the shell fails to hydrate, its violations vanish and the tally falls — the gate passes, having measured less of the application. Guard it by asserting the route count and failing the run when any route throws, and treat a mysteriously improved number with the same suspicion as a worsened one.
  • Lazy content below the fold. Images and cards that load on scroll produce findings on a fast runner and none on a slow one, which shows up as a minor count that drifts by two or three. Either scroll to the bottom of each route before scanning, or exclude the lazily rendered container from the scan context and note it in the ceilings file’s note field so the exclusion is not forgotten.

Pipeline Impact

Wire it as one job with two steps, and give the enforcement step a five-minute timeout so a hung route cannot occupy a runner for the job’s full allowance. The recording step is not in the workflow at all — it is run by hand, once, and its output is committed.

name: a11y-impact-ceilings
on:
  pull_request:
    paths:
      - 'src/**'
      - 'a11y/**'
      - '.github/workflows/a11y-impact-ceilings.yml'
jobs:
  impact-ceilings:
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npm run build
      - name: Serve the production build
        run: npx serve -s dist -l 4173 &
      - name: Enforce impact ceilings
        id: gate
        timeout-minutes: 5
        run: node a11y/impact-gate.mjs --enforce | tee ceilings-report.md
      - name: Put the table in the job summary
        if: always()
        run: cat ceilings-report.md >> "$GITHUB_STEP_SUMMARY"

tee matters: the script writes its table to stdout, so piping through tee keeps the log readable and leaves a file to append to the job summary. Because set -o pipefail is not on by default in the Actions shell, add it if the exit code of the script rather than of tee must fail the step — or simply put the summary append in its own if: always() step as above, which is why the exit code survives here.

For the first three days, add --warn-only to the enforcement command and leave the check optional in branch protection. Watch which pull requests would have been blocked; every one of them is either a real regression or a piece of scan instability worth fixing before the gate becomes mandatory — the discipline for that observation window is set out in soak-testing a new accessibility gate in warning mode. Once a full working week passes with no spurious blocks, drop the flag and promote the job to a required status check, which is the mechanical step described in requiring accessibility status checks in branch protection and the point at which the four numbers start to fall.

Ten-day rollout from recording to a required check Day zero records the ceilings by hand. Days one to three run the gate with the warn-only flag as an optional check, during which one real regression and one flaky lazy-image count are found. From day four the flag is removed and the job becomes a required status check. Rollout: optional first, required once the tally is proven stable record by hand, once --warn-only, optional check nothing is blocked yet blocking, required status check ceilings can only fall from here day 0 day 1 day 3 day 4 day 10 real regression caught critical 4 → 5, fixed today flaky minor count fixed scroll before scanning Both problems surfaced while the gate could not block anyone — the point of the warn-only window.
The warning window exists to find scan instability, not to postpone the decision — three days is usually enough, and past a week the gate is being avoided rather than validated.

Common Pitfalls

  • Recording the ceilings from a feature branch, so the very first pull request that merges main into it reports the difference between the two branches as a regression.
  • Counting results.violations.length instead of node counts, which makes a page go from one broken input to nine with no change in the gate at all.
  • Committing the ceilings file without the recordedFrom provenance, leaving nobody able to tell whether 212 was measured against 38 routes or 41.
  • Lowering a ceiling in the same pull request that fixes the findings, which is well intentioned but makes the fix’s own CI run fail if any single finding is flaky — lower it in a follow-up once the new number has held for a week.
  • Leaving the gate on --warn-only indefinitely because nobody wants to be the person who makes it blocking; a warning-only gate does not stop the count from growing, it only documents that it grew.

FAQ

Why four impact levels rather than one total? A single total can be satisfied by a trade: delete two minor findings, add one critical, and the total falls while the product gets worse for someone using a screen reader. Four independent comparisons make that trade fail, because the critical row rises regardless of what happens to the others. It is still coarser than a per-rule budget, but it closes the worst of the loophole for the price of three extra integers.

When should the ceilings be lowered, and by whom? By hand, in a small pull request, once a level has read below its ceiling on the default branch for a full week — long enough that a lucky run cannot set a floor the branch cannot reach again. Doing it manually is fine for the first month or two; the moment the team forgets twice in a row, replace the ritual with the scheduled job described in the parent guide, which computes the new floor from recent history and opens the pull request itself.

What happens when axe-core is upgraded? Counts move, usually upward, because a minor release adds rules and occasionally reclassifies an impact. Upgrade in an isolated pull request with --warn-only, read the table, and if the rise is genuinely the new rules rather than a regression, re-record the ceilings in that same pull request with the axe version in recordedFrom. Never mix a scanner upgrade with a feature change, or the table becomes unreadable.

Back to Progressive Threshold Management