Blocking Pull Requests on Critical Accessibility Violations

A gate is a script that answers one question — does this change ship? — and every property that makes it trustworthy is in how it answers, not in whether it can. This guide is part of GitHub Actions a11y Pipeline Setup, and it covers the four decisions that separate a gate teams keep from a gate teams delete: which impacts block, whether the count is of rules or of failing nodes, what the failure message says, and how an exemption is granted so that it expires instead of accumulating.

Root Cause

The scanner’s own exit code is the wrong gate because it encodes a policy nobody chose. axe-core marks every result with an impact of minor, moderate, serious or critical, and the CLI’s --exit flag fails on any of them, so the first pull request that trips a moderate landmark advisory blocks a release. The team’s response is predictable and permanent: the flag comes off, the job becomes advisory, and the pipeline now catches nothing. Choosing the blocking set explicitly — critical alone at first, critical plus serious once the backlog is clear — is what makes the gate defensible in the conversation where somebody wants it removed.

The second problem is arithmetic. An axe report is a list of violations, one per rule per page, each carrying a nodes array of the elements that failed it. A cart page that renders twenty-four rows with an unlabelled icon button produces exactly one entry in violations, so a gate that reports violations.length says “1 problem” about twenty-four broken controls, and a threshold expressed in violations treats that page as equivalent to one missing form label. Counting nodes changes both the number and the priority order, and it is the count that correlates with how much a user actually encounters the barrier.

The third is that the failure message is the product. A gate that prints Accessibility check failed: 3 violations and exits 1 sends the author to the run log, then to the artifact, then to a JSON file, and each hop loses people. A gate that prints the rule ID, the impact, the route and the exact selector for every failing node lets the author fix the problem from the pull-request page. The last is exemptions. Disabling a rule in the axe configuration, or adding it to a global ignore list, is invisible six weeks later: nobody knows who disabled it, for which element, or when it was supposed to come back. An exemption that carries a rule ID, a selector, an owner and an expiry date — enforced by the same script that reads it — is a scheduled repayment rather than a silent write-off.

Four violations, thirty-seven failing nodes Each of the four bars is a single entry in the violations array, but the bar heights show the number of failing nodes behind each entry: twenty-four for button-name, nine for colour-contrast, three for aria-allowed-attr and one for label. One report, counted two ways failing nodes 20 10 0 24 9 3 1 button-name color-contrast aria-allowed-attr label 1 violation 1 violation 1 violation 1 violation violations: 4 nodes: 37
The left-hand bar is one entry in the violations array and twenty-four broken controls; any threshold expressed in violations cannot tell those apart.

Configuration

The gate is a single Node script with no dependencies, no network access and a read-only token, so it can be run identically on a laptop and in the job. It takes the directory of merged shard reports, flattens them to node-level findings, applies the allowlist, and prints one line per blocking node before it exits.

// scripts/a11y/gate.mjs — usage: node scripts/a11y/gate.mjs a11y-out
import { readdirSync, readFileSync, existsSync } from 'node:fs';
import { join } from 'node:path';

const dir = process.argv[2] ?? 'a11y-out';
// --expect-shards N turns a scan job that died before writing into a failure.
const expectIndex = process.argv.indexOf('--expect-shards');
const expectShards = expectIndex > -1 ? Number(process.argv[expectIndex + 1]) : 0;
// Start at critical only; widen to serious once the backlog is empty.
const BLOCKING = new Set((process.env.A11Y_BLOCK_ON ?? 'critical').split(','));
// warn mode prints everything and still exits 0, for the soak period.
const WARN_ONLY = process.env.A11Y_MODE === 'warn';
const today = new Date().toISOString().slice(0, 10);

// 1. Flatten every shard into one node-level list. A violation with 24 nodes
//    becomes 24 findings, because 24 controls are broken.
const findings = [];
const shards = readdirSync(dir).filter((f) => f.startsWith('axe-'));
if (expectShards && shards.length !== expectShards) {
  console.error(`BLOCK expected ${expectShards} shard file(s), found ${shards.length}`);
  process.exit(1);
}
for (const file of shards) {
  const shard = JSON.parse(readFileSync(join(dir, file), 'utf8'));
  for (const page of shard.results) {
    for (const violation of page.violations) {
      for (const node of violation.nodes) {
        findings.push({
          rule: violation.id,
          impact: violation.impact,
          help: violation.help,
          route: page.route,
          width: shard.width,
          selector: node.target.join(' '),
        });
      }
    }
  }
}

// 2. Load the allowlist and reject malformed or expired entries outright.
const allowPath = 'a11y/allowlist.json';
const allowlist = existsSync(allowPath)
  ? JSON.parse(readFileSync(allowPath, 'utf8'))
  : [];
const REQUIRED = ['rule', 'selector', 'owner', 'expires', 'reason'];
const schemaErrors = [];
const expired = [];
for (const [index, entry] of allowlist.entries()) {
  const missing = REQUIRED.filter((k) => !entry[k]);
  if (missing.length) {
    schemaErrors.push(`entry ${index}: missing ${missing.join(', ')}`);
    continue;
  }
  if (!/^\d{4}-\d{2}-\d{2}$/.test(entry.expires)) {
    schemaErrors.push(`entry ${index}: expires must be YYYY-MM-DD`);
  } else if (entry.expires < today) {
    expired.push(entry);
  }
}

// 3. Partition the findings. An exemption matches on rule AND selector, never
//    on rule alone, so exempting one badge cannot exempt a whole page.
const live = allowlist.filter((e) => e.expires >= today && !e.__invalid);
const blocked = [];
const exempt = [];
for (const f of findings) {
  if (!BLOCKING.has(f.impact)) continue;
  const hit = live.find((e) => e.rule === f.rule && e.selector === f.selector);
  (hit ? exempt : blocked).push(f);
}

// 4. Report. Every blocking line names the rule, the impact and the selector.
for (const f of blocked) {
  console.error(
    `BLOCK ${f.rule} (${f.impact}) ${f.route} @${f.width}px -> ${f.selector}`,
  );
  console.error(`      ${f.help}`);
}
for (const e of expired) {
  console.error(
    `BLOCK expired exemption ${e.rule} on ${e.selector} ` +
      `(owner ${e.owner}, expired ${e.expires})`,
  );
}
for (const msg of schemaErrors) console.error(`BLOCK allowlist ${msg}`);
console.log(
  `scanned ${shards.length} shard(s): ${findings.length} finding(s), ` +
    `${blocked.length} blocking node(s), ${exempt.length} exempt, ` +
    `${expired.length} expired exemption(s)`,
);

const failures = blocked.length + expired.length + schemaErrors.length;
if (failures && !WARN_ONLY) process.exit(1);

The allowlist is a reviewed file in the repository, not a configuration flag. Four of its five fields exist to make the exemption self-cancelling: the selector keeps it narrow, the owner makes it assignable, the expiry makes it temporary, and the reason makes it reviewable by somebody who was not in the original conversation. Ninety days is a workable default — long enough to schedule a fix into a sprint, short enough that nobody treats it as permanent.

[
  {
    "rule": "color-contrast",
    "selector": ".promo-badge > span",
    "owner": "@growth-web",
    "expires": "2026-08-02",
    "reason": "Brand palette review scheduled; badge is decorative in FY26 designs."
  },
  {
    "rule": "aria-allowed-attr",
    "selector": "#chat-widget iframe",
    "owner": "@platform-support",
    "expires": "2026-09-15",
    "reason": "Third-party chat markup; vendor ticket SUP-4192 tracks the fix."
  }
]
An exemption that cancels itself A ninety-day band shows an exemption honoured from the day it is added, entering a warning window fourteen days before expiry, and blocking the gate once the expiry date has passed. Beneath it, the five required fields and the failure message the gate prints. Ninety days from agreement to enforcement added 2026-05-04 T-14 warning expires 2026-08-02 exemption honoured warning window gate fails rule: color-contrast selector: .promo-badge > span owner: @growth-web expires: 2026-08-02 · reason: set on 2026-08-03 the gate prints BLOCK expired exemption color-contrast on .promo-badge owner @growth-web
The expiry is enforced by the same script that honours the exemption, so nobody has to remember to review the file.

The workflow step is deliberately thin, because everything that could need debugging lives in the script rather than in YAML. The gate is the only job whose exit code matters, and it asserts the shard count so a scan job that died before writing a file cannot produce a green result by omission.

      - name: Enforce the accessibility budget
        run: node scripts/a11y/gate.mjs a11y-out --expect-shards 6
        env:
          # Widen to 'critical,serious' after the soak period ends.
          A11Y_BLOCK_ON: critical
          A11Y_MODE: ${{ github.ref == 'refs/heads/main' && 'warn' || 'block' }}
How the gate decides on a single failing node Four questions in order: is the impact in the blocking set, is there an allowlist entry matching both rule and selector, does that entry carry an owner and an expiry, and is the expiry still in the future. Each no branches right to an outcome, and the final yes allows the node as tracked debt. impact in the blocking set? allowlist entry for rule + selector? entry has owner, expiry, reason? expiry date still in the future? reported in the summary, not blocking block: print rule, impact, selector block: allowlist entry rejected block: exemption expired, name owner allowed, counted as tracked debt no no no no yes yes yes yes
Three of the four block outcomes have nothing to do with the scan: a malformed or lapsed exemption fails the gate exactly like an unfixed violation.

Validation

Download a real report from a previous run and drive the script against it. Because the gate takes a directory and reads its environment, every branch is reachable from a shell without pushing anything.

gh run download 1849321 --dir a11y-out --pattern 'a11y-*'

# 1. Baseline: critical only, current allowlist.
A11Y_BLOCK_ON=critical node scripts/a11y/gate.mjs a11y-out; echo "exit=$?"
# scanned 6 shard(s): 37 finding(s), 24 blocking node(s), 0 exempt, 0 expired
# exit=1

# 2. The message names what to fix, one line per broken control:
A11Y_BLOCK_ON=critical node scripts/a11y/gate.mjs a11y-out 2>&1 | head -3
# BLOCK button-name (critical) /checkout/cart @375px -> #cart li:nth-child(3) > button
#       Buttons must have discernible text
# BLOCK button-name (critical) /checkout/cart @375px -> #cart li:nth-child(4) > button

# 3. Prove the expiry is enforced: backdate an entry and re-run.
node -e "const f='a11y/allowlist.json',fs=require('fs');
const a=JSON.parse(fs.readFileSync(f));a[0].expires='2026-01-01';
fs.writeFileSync('/tmp/expired.json',JSON.stringify(a));"
cp a11y/allowlist.json /tmp/keep.json && cp /tmp/expired.json a11y/allowlist.json
node scripts/a11y/gate.mjs a11y-out 2>&1 | grep expired
# BLOCK expired exemption color-contrast on .promo-badge > span
#       (owner @growth-web, expired 2026-01-01)
cp /tmp/keep.json a11y/allowlist.json

# 4. Warning mode still prints everything and exits 0.
A11Y_MODE=warn node scripts/a11y/gate.mjs a11y-out > /dev/null 2>&1; echo "exit=$?"
# exit=0

The exit-code contract is worth writing down, because branch protection reacts only to the number and reviewers need to know what each one means. Reserving distinct codes for “the gate found violations” and “the gate could not run” is the distinction argued for in choosing exit codes for warning and blocking a11y jobs.

Situation Blocked nodes Exit code Check status
No findings at blocking impact 0 0 pass
Only moderate and minor findings 0 0 pass
Every critical node covered by a live exemption 0 0 pass, debt logged
One critical node uncovered 1+ 1 fail, selector printed
Exemption past its expiry date 0 1 fail, owner named
Allowlist entry missing owner 0 1 fail, schema error
Fewer shard files than expected n/a 1 fail, scan incomplete

Edge Cases and Conditional Guards

  • Selectors that churn: an exemption keyed on #cart li:nth-child(3) > button stops matching the moment a row is inserted, and the gate blocks. That is the correct behaviour rather than a bug — prefer a stable hook such as [data-testid="promo-badge"] in the exemption, and treat a suddenly-unmatched entry as a signal that the markup moved and the exemption needs re-reviewing.
  • Incomplete results are never blocking: axe puts nodes it could not decide about into incomplete, which includes anything inside a closed shadow root or behind aria-busy="true". The gate reads violations only. Surfacing incomplete for triage belongs in the reporting job described in annotating pull requests with axe-core violation comments, never in the exit code.
  • Virtualised and repeated content inflates node counts: a broken cell in a virtualised table reports as many nodes as happen to be rendered, so the same defect yields 12 nodes on one runner and 20 on a faster one. Deduplicate by rule plus component before counting if the number feeds a budget, and keep the raw node list for the message; ratcheting a numeric budget under that kind of variance is covered in progressive threshold management.

Pipeline Impact

This script is the only thing in the workflow permitted to fail the run, which is why the reporting job carries continue-on-error: true and the scan job exits zero unless the browser itself crashed. One deciding job means one status check to register in branch protection and one place to look when the decision is wrong; the registration mechanics are in requiring accessibility status checks in branch protection.

Roll it out in warning mode. A11Y_MODE=warn prints every blocking line and exits zero, so a week of real pull requests reveals the true violation surface before anybody is stopped by it, and the allowlist gets populated by evidence rather than guesswork. When the mode flips, the diff is one line in the workflow and the behaviour is already familiar. Skipping the soak is how a gate acquires a reputation for blocking merges over pre-existing debt in its first week, and reputations like that outlive the configuration change that caused them.

Common Pitfalls

  • Reporting violations.length in the failure message, so twenty-four broken buttons and one missing label both read as “1 violation”.
  • Blocking on minor and moderate from day one, which produces a gate that fails on advisory findings and gets switched off within a fortnight.
  • Disabling a rule in the axe configuration instead of adding a scoped exemption, which turns off the check for the entire application with no record of why.
  • Writing an exemption keyed on the rule ID alone, so one exempted decorative badge silently exempts every contrast failure on every page.
  • Allowing an exemption with no expiry date, which converts a two-week workaround into a permanent hole nobody can attribute.
  • Letting a missing shard file pass as zero findings, so a crashed scan job reads as a clean bill of health.
  • Printing the raw JSON report to the log as the failure message, which is technically complete and practically unreadable.
  • Filtering only the newest shard when the matrix has several, so a failure at 375 pixels never blocks because the 1440-pixel shard was clean.

FAQ

Should serious block as well as critical? Eventually, yes — several of the failures that most reliably stop a screen-reader user, including missing form labels and broken ARIA references, are classified serious rather than critical. The sequencing matters more than the destination: block on critical until the existing critical count is zero and stays there for a few sprints, then widen the set in a single change with the allowlist prepared in advance, so the first run after the change fails for reasons the team has already seen.

How does the allowlist differ from a baseline file of known violations? A baseline is a snapshot: it records everything currently failing and lets all of it through until somebody regenerates it, which means it grows silently and never expires. An allowlist is a set of individually argued exceptions, each with an owner, a reason and a date, and the gate fails when one lapses. A baseline answers “what was broken when we turned this on”; an allowlist answers “who agreed to this, and until when”.

What if a violation is a genuine false positive? Then it does not belong in the allowlist at all, because the entry would expire and re-block a non-problem. Fix it at the scanner level — a scoped exclude for a third-party container, or a rule option that matches the real markup contract — and record why in the axe configuration, as described in reducing false positives in automated accessibility scanners. Keep the allowlist for real barriers that are real, acknowledged and scheduled.

Back to Github Actions a11y Pipeline Setup