Testing Internationalized Labels in Automated a11y Workflows

An icon-only button with aria-label="Close" passes every accessibility scanner in the world, in every locale, forever. This guide is part of Internationalization & Localization Testing, and it covers the one assertion that closes that hole: proving that each accessible name on the page is the translated value from the message catalogue for the locale under test, and that no accessible name reached the DOM without passing through the translation layer at all. Both halves are needed, because a dropped catalogue key and a string hardcoded in a template are different defects with different detectors.

Root Cause

Every name-related rule in axe-core is an existence check. button-name, image-alt, input-image-alt, link-name, and label all resolve the accessible name for a node and pass if the result is non-empty. There is no dictionary of expected strings, no language detection on the name, and no notion of which locale the page is meant to be rendering. A German page whose close button announces “Close” is structurally perfect and semantically broken, and the scanner reports zero violations. WCAG 2.2 SC 4.1.2 (Name, Role, Value) is satisfied by the existence of a name; the mismatch between the name’s language and the page’s declared language is a SC 3.1.2 (Language of Parts) problem that no automated rule can see without external knowledge.

The failure mode that produces those English names is silent by design. Every mainstream i18n runtime falls back to the source locale when a key is missing from the target catalogue, because rendering a raw key or an empty string in production is worse than rendering the wrong language. That fallback is correct behaviour for the runtime and catastrophic for testing, because the exact defect the test needs to catch is the thing the runtime is engineered to hide. Visual QA does not catch it either: aria-label, alt, title, and visually hidden helper text never appear on screen, so a reviewer clicking through the German build sees a fully translated page.

The second defect never touches the catalogue. Extraction tooling scans source for t('some.key') calls and builds the source catalogue from what it finds; a string written inline as aria-label="Close" in a template is invisible to the extractor, so it never appears in the source catalogue, never reaches a translator, and never shows up as missing in any coverage report. Catalogue-based assertions cannot detect it, because from the catalogue’s point of view the string does not exist. Detecting it requires inverting the test: render the app in a locale where every translated string is visibly marked, and treat any unmarked accessible name as proof that the string bypassed the translation layer.

Classifying an accessible name found in the DOM A name present in the target locale catalogue is translated; a name present only in the source catalogue is a fallback from a dropped key; a name in neither catalogue was hardcoded in a template. accessible name in the tree in ar catalogue? yes translated assertion passes no in en catalogue? yes dropped key source-locale fallback no hardcoded in a template
The two red-flag branches need different detectors: catalogue parity finds the dropped key, and only a pseudo-locale run finds the hardcoded string.

Configuration

Stage one runs in Node with no browser and answers one question: does every key that can produce an accessible name exist, non-empty and distinct from the source value, in every blocking locale? Marking which keys those are is the only convention the codebase has to adopt. A leaf-name convention works well and needs no new tooling: any key whose final segment is label, alt, title, hint, or announce is name-producing.

// scripts/assert-name-coverage.mjs — fails before any browser starts
import { readFile, writeFile } from 'node:fs/promises';

const SOURCE = 'en-US';
const TARGETS = (process.env.BLOCKING_LOCALES ?? 'de-DE,fr-FR,ja-JP,ar-SA').split(',');
const NAME_LEAVES = new Set(['label', 'alt', 'title', 'hint', 'announce']);
// Strings that are legitimately identical across locales: brands, units, file formats.
const IDENTICAL_OK = new Set(['common.brand.label', 'export.pdf.label', 'units.km.label']);

const flat = (o, p = '') =>
  Object.entries(o).flatMap(([k, v]) =>
    v && typeof v === 'object' ? flat(v, `${p}${k}.`) : [[`${p}${k}`, String(v)]]);

const load = async (tag) => new Map(flat(JSON.parse(await readFile(`locales/${tag}.json`, 'utf8'))));

const source = await load(SOURCE);
// Only name-producing keys are gated; body copy is a translation-team concern, not a11y.
const nameKeys = [...source.keys()].filter((k) => NAME_LEAVES.has(k.split('.').pop()));

const findings = [];
for (const tag of TARGETS) {
  const target = await load(tag);
  for (const key of nameKeys) {
    const value = target.get(key);
    if (value === undefined) findings.push({ tag, key, kind: 'missing' });
    else if (value.trim() === '') findings.push({ tag, key, kind: 'empty' });
    else if (value === source.get(key) && !IDENTICAL_OK.has(key))
      findings.push({ tag, key, kind: 'identical-to-source' });
  }
}

await writeFile('reports/name-coverage.json', JSON.stringify({ nameKeys: nameKeys.length, findings }, null, 2));
for (const f of findings) console.error(`${f.kind}: ${f.key} [${f.tag}]`);
console.log(`checked ${nameKeys.length} name keys across ${TARGETS.length} locales`);
// Distinct exit codes so the workflow can annotate differently per failure class.
process.exit(findings.some((f) => f.kind === 'missing') ? 2 : findings.length ? 3 : 0);

The leaf-name convention is a deliberate trade-off. It costs nothing to adopt, needs no new metadata format, and is enforceable by a lint rule that rejects an aria-label bound to a key whose leaf is not in the set. Its weakness is that it depends on naming discipline: a key called cart.close.text that ends up in an aria-label is invisible to the gate. Two backstops close that gap. Keep an explicit override list of extra keys to treat as name-producing, and assert a floor on the number of keys the script found, so a rename that silently shrinks the gated set fails rather than passing quietly. Larger codebases eventually graduate to an explicit flag in the catalogue — a sibling "__a11y": true node or a separate a11y-keys.json manifest — which removes the naming dependency at the cost of a format the extraction tooling has to preserve.

Stage two needs a locale that makes translated strings self-identifying. Generate a pseudo-locale from the source catalogue by wrapping every value in sentinel brackets and substituting accented look-alikes for ASCII letters. The accents make text expansion and font-fallback problems visible at the same time; the brackets are what the assertion actually tests.

// scripts/build-pseudo-locale.mjs — writes locales/en-XA.json
import { readFile, writeFile } from 'node:fs/promises';

const ACCENTS = { a: 'á', c: 'ç', e: 'é', i: 'í', o: 'ó', s: 'š', u: 'ü', y: 'ý', z: 'ž',
  A: 'Á', C: 'Ç', E: 'É', I: 'Í', O: 'Ó', S: 'Š', U: 'Ü', Y: 'Ý', Z: 'Ž' };

const pseudo = (value) => {
  // Leave {placeholders} and <tags> untouched so interpolation still resolves.
  const parts = value.split(/(\{[^}]*\}|<[^>]*>)/g);
  const body = parts
    .map((p, i) => (i % 2 ? p : [...p].map((ch) => ACCENTS[ch] ?? ch).join('')))
    .join('');
  return `[!!${body}!!]`;   // sentinel markers: the assertion looks for these
};

const walk = (node) =>
  Object.fromEntries(Object.entries(node).map(([k, v]) =>
    [k, v && typeof v === 'object' ? walk(v) : pseudo(String(v))]));

const source = JSON.parse(await readFile('locales/en-US.json', 'utf8'));
await writeFile('locales/en-XA.json', JSON.stringify(walk(source), null, 2));

The test then reads every accessible name from the page and requires the markers. Playwright’s ARIA snapshot is the cleanest source for this, because it yields exactly the role-and-name pairs the accessibility tree exposes, including open shadow roots — the same mechanism described in the guide on asserting accessibility tree names with Playwright snapshots.

// tests/i18n-a11y/pseudo-locale.spec.ts
import { test, expect } from '@playwright/test';

// Names that legitimately never pass through t(): fixture data and formatted values.
const NON_TRANSLATABLE = [/^\d+([.,]\d+)?$/, /^Acme Widget \d+$/, /^UTC$/];

test.use({ locale: 'en-XA' });

test('every accessible name carries the pseudo-locale markers', async ({ page }) => {
  await page.goto('/en-XA/checkout');
  await page.waitForFunction(() => document.documentElement.dataset.i18n === 'ready');

  const snapshot = await page.locator('body').ariaSnapshot();
  // Each snapshot line looks like: - button "[!!Çĺóšé!!]"
  const names = [...snapshot.matchAll(/^\s*- \w+ "([^"]+)"/gm)].map((m) => m[1]);
  expect(names.length, 'aria snapshot produced no named nodes').toBeGreaterThan(10);

  const hardcoded = names.filter(
    (n) => !n.includes('[!!') && !NON_TRANSLATABLE.some((re) => re.test(n))
  );
  expect(hardcoded, `strings that never reached the catalogue: ${hardcoded.join(' | ')}`).toEqual([]);
});
Pseudo-locale marker verdicts Two catalogue-sourced names render wrapped in sentinel brackets with accented letters and pass, while a name written inline in a template renders bare and is reported as hardcoded. source of the string accessible name in en-XA verdict t('search.submit.label') catalogue value: Search [!!Šéárçh!!] translated t('cart.close.label') catalogue value: Close cart [!!Çĺóšé çárt!!] translated aria-label="Submit" in JSX never extracted, no key Submit hardcoded no sentinel brackets means the string never passed through the translation layer
The bracket markers turn "is this string translatable" into a regex test on the accessible name, which is why the pseudo-locale finds what catalogue parity cannot.

Validation

Confirm each detector by breaking the thing it is supposed to catch. Delete one name-producing key from the German catalogue and run stage one; it should name the key and the locale and exit 2 without launching a browser. Then add an inline aria-label to a component and run the pseudo-locale spec; it should list the bare string.

# 1. Prove the catalogue detector fires on a dropped key.
jq 'del(.cart.close.label)' locales/de-DE.json > tmp.json && mv tmp.json locales/de-DE.json
node scripts/assert-name-coverage.mjs; echo "exit=$?"
# missing: cart.close.label [de-DE]
# checked 148 name keys across 4 locales
# exit=2

# 2. Prove the pseudo-locale detector fires on a hardcoded string.
node scripts/build-pseudo-locale.mjs
npx playwright test tests/i18n-a11y/pseudo-locale.spec.ts --reporter=list
#   ✘  every accessible name carries the pseudo-locale markers
#      strings that never reached the catalogue: Submit
#      Expected: []   Received: ["Submit"]

The checked 148 name keys line matters as much as the failure. If that number drops between runs, the leaf-name convention has stopped matching the codebase — someone renamed label to text — and the gate is quietly checking less than it did. Assert a floor on it in CI so shrinking coverage is itself a failure.

Edge Cases and Conditional Guards

  • Names sourced from data, not the catalogue. Product titles, user names, and formatted numbers legitimately have no key, so they will always appear bare in the pseudo-locale run. Seed the test with deterministic fixture data and express the exemptions as narrow regular expressions, or mark the subtree with data-user-content and drop those nodes from the snapshot before asserting — never widen the exemption to a whole page.
  • Interpolated names. A value like Remove {item} from cart becomes [!!Rémóvé!!]{item}[!! fróm çárt!!] only if the generator splits on placeholders, and the rendered name then contains untransformed data in the middle. Test with name.includes('[!!') for those keys rather than anchoring the pattern to both ends of the string.
  • Names that only exist after interaction. An ARIA snapshot of the initial page never sees the labels inside a closed dialog, a collapsed disclosure, or an unmounted route, so coverage silently stops at the first screen. Drive the walk through a route list and open each disclosure and dialog before snapshotting, and keep a build-time extraction lint as the backstop for surfaces the test never reaches.

Pipeline Impact

Keep the two stages in separate jobs, because their costs differ by two orders of magnitude. Stage one needs npm ci and nothing else — no browser download, no server boot — so it reports in roughly twenty seconds and is the fastest accessibility signal in the pipeline. Wiring it as its own required check means a dropped translation key fails the pull request before the browser matrix has finished installing Chromium.

Stage two is a browser job and belongs outside the shipped-locale matrix. The pseudo-locale is a diagnostic build whose deliberately mangled strings will trip contrast and clipping checks, so running it alongside real locales pollutes those results. Give it its own job, disable the visual checks in that project, and gate it on the hardcoded-string assertion only. Generate locales/en-XA.json during the job rather than committing it; a committed pseudo-locale drifts out of sync with the source catalogue within a sprint and starts reporting stale markers, and it inflates every translation-tool diff with a file no human will ever read.

One more scheduling decision is worth making explicitly: whether stage two runs on every pull request or on merge to the main branch. Hardcoded strings appear when a component is written, not when copy changes, so the defect rate correlates with new component code rather than with the diff surface most pull requests touch. Running the walk on pull requests that modify component directories, and on a nightly schedule otherwise, keeps the signal without paying four minutes on every copy-only change.

Exit codes carry the failure class: 2 for a missing key, 3 for an empty or source-identical value, 1 for a hardcoded string from the Playwright job. That split lets the workflow annotate a missing key as blocking while treating a source-identical value as a warning during a translation cycle — the tiering pattern set out in the guide on choosing exit codes for warning and blocking a11y jobs. Upload reports/name-coverage.json as an artifact and render it into a pull-request comment using the output shape described in the guide on structuring JSON violation output for Slack and GitHub annotations, so a reviewer sees the offending key and locale without opening a log.

Cost of the two-stage name-coverage gate Catalogue parity finishes in under twenty seconds and can fail the pull request before any browser starts, while the pseudo-locale browser stage takes about four minutes. stage 1 catalogue parity, no browser 19 s missing key: exit 2 before a browser launches stage 2 all keys present pseudo-locale browser walk, 3 min 49 s 0 s 20 s 4 min 8 s wall-clock cost of the two name-coverage stages on one pull request
Ordering the cheap catalogue check first means the most common defect — a dropped key — costs twenty seconds of CI rather than four minutes.

Common Pitfalls

  • Asserting that accessible names are non-empty and calling that internationalization coverage; every source-language fallback passes that assertion.
  • Gating on the whole catalogue instead of the name-producing subset, which buries a missing aria-label key under hundreds of untranslated marketing sentences.
  • Skipping the source-identical check, so a translation memory that copied the English string through untouched reports as fully covered.
  • Shipping the pseudo-locale in SHIPPED_LOCALES, which floods the real locale matrix with contrast and clipping failures caused by the deliberately mangled strings.
  • Broadening the non-translatable exemption list to whole components rather than specific string patterns, which quietly re-opens the hole the pseudo-locale was added to close.
  • Running the pseudo-locale walk only against the landing page, so every label behind a dialog or a lazy route stays unverified.

FAQ

Why not just detect the language of the accessible name instead of comparing to the catalogue? Language detection is unreliable on the short strings that accessible names actually are. “OK”, “Menu”, “Email”, and most product nouns are indistinguishable across a dozen languages, and any detector confident enough to flag them will also flag correctly translated loanwords. Comparing against the catalogue value replaces a probabilistic guess with an exact expectation, and the catalogue is already the artifact translators edit.

Does the pseudo-locale replace the catalogue parity check? No, they catch disjoint defects. The pseudo-locale is built from the source catalogue, so a key that exists in the source and is missing from German renders perfectly marked in en-XA and the browser walk passes. Conversely a hardcoded template string never appears in any catalogue, so parity has nothing to compare. Both stages are needed, and the cheap one should run first.

How does this interact with suppression lists used to reduce scanner noise? It does not overlap with them. Suppression lists tune structural rules that produce false positives, as covered in the guide on reducing false positives in automated accessibility scanners, whereas these assertions compare exact strings and have no heuristic component. A failure here is always either a real dropped key, a real hardcoded string, or a stale exemption pattern, so the finding should be fixed rather than suppressed.

Back to Internationalization Localization Testing