Comparing Playwright and Cypress for WCAG Compliance Testing

Both runners execute the same scanning engine, so a violation either runner finds is a real violation. The decision is about what each one can reach, how fast it can reach it sixty times, and how much of the evidence survives into a report. This guide is part of Playwright Accessibility Plugin Integration, and it compares the two runners on the four axes that actually change accessibility outcomes — frame and tab reach, violation reporting, parallelism granularity, and determinism under retry — then gives a recommendation and a realistic migration cost for a team that already owns one suite.

Root Cause

The usual Playwright-versus-Cypress argument is about developer experience: time-travel debugging against traces, async/await against a command queue, how pleasant the assertion syntax is. For functional testing that argument is genuinely close, and either runner is defensible. Accessibility scanning stresses none of those axes. A scan is not a click — it is a whole-tree evaluation that has to execute inside every scriptable document on the page, produce the same answer twice in a row, and repeat sixty times inside a pull-request time budget. The two runners diverge sharply on exactly those three requirements, which is why the accessibility answer can differ from the functional one.

The divergence starts with where the test code runs. Cypress executes the spec inside the browser, in a privileged parent document, and loads the application under test into a sibling iframe. Anything the spec touches has to be reachable from that iframe’s document, cy.injectAxe() puts the engine into that document, and cy.checkA11y() runs it there. A nested same-origin iframe is fine — the engine walks into it. A nested cross-origin iframe is not, because the spec has no execution context inside it and the browser will not grant one. Playwright inverts the topology: the test code runs in a Node process outside the browser and drives it over the DevTools protocol, so a BrowserContext owns pages, each page owns a frame tree, and the builder injects into the main frame and then into every child frame it can attach to — including cross-origin ones, because CDP grants an execution context per frame regardless of origin.

That difference is invisible in the results. A checkout page with a payment form served from a payment provider’s domain produces a shorter violation list under Cypress than under Playwright, and neither run says “one document was not scanned” unless you look for it. The engine records an incomplete entry for its frame-testing check rather than a violation, and a suite that asserts only on violations reads an unscanned third of the page as clean. Multi-tab work has the same shape: a consent screen opened with window.open, a print preview, a terms-and-conditions popup are all real documents with real obligations under WCAG 2.2, and in Cypress they are typically reached by stubbing window.open so the URL loads in the same frame — which tests a page that no user ever sees in that layout. In Playwright a popup arrives as a first-class Page from context.waitForEvent('page') and a fresh builder scans it directly.

Where the test code runs, and what it can reach On the left, Cypress spec code lives in a runner document with the application in a sibling iframe; a nested same-origin frame is scanned and a nested cross-origin frame is blocked. On the right, a Node process drives a BrowserContext over the DevTools protocol and reaches the main frame, a cross-origin payment frame and a second tab. Cypress: spec runs in the browser Playwright: spec runs in Node runner document holds the spec code application iframe cy.injectAxe() lands here same-origin child frame scanned cross-origin child frame no context, reported incomplete Node process drives CDP BrowserContext Page 1 main frame payment frame cross-origin Page 2 popup tab scanned An unreachable document produces a shorter violation list, not an error, so the gap stays invisible.
Reach is a property of the execution topology, not of the scanning engine, so the same page yields different violation counts in the two runners.

Reporting is where the daily cost lives. cy.checkA11y() fails the test through the command log; the violation detail reaches a human only if a violation callback is wired to a cy.task that prints it, or if the run is recorded to a dashboard that keeps the log. The failure screenshot captures the application iframe, which is useful, but the rule id and the failing node’s selector are not in the artifact by default. Playwright’s model attaches arbitrary blobs to a test result, so the complete result object, a tab-separated digest and a cropped screenshot of the offending node all land in the HTML report beside the trace. That is a triage-speed difference rather than a capability one — a determined Cypress setup can write the same files — but it is a difference that repeats on every red build.

Parallelism granularity is the axis with the biggest wall-clock consequence. Cypress distributes work at spec-file granularity: one machine runs one spec file at a time, and load balancing happens across machines through an orchestration service. Sixty route scans written as a for loop inside one spec file therefore get no parallelism at all, and the fix is a repository-shape change — sixty spec files — forced by the runner rather than chosen. Playwright distributes at test granularity across worker processes on a single machine, and shards across machines on top of that, so the for loop parallelises as written. On a four-core runner with scans averaging 2.5 seconds, sixty scans take about 150 seconds serially, about 45 seconds across four local workers, and need four separate machines plus sixty spec files to reach a comparable number in Cypress.

Wall clock for sixty route scans Four columns compare strategies: one hundred fifty seconds serial on a single worker, forty-five seconds with four Playwright workers on one machine, fifty-two seconds with Cypress spec-level distribution across four machines, and one hundred fifty-eight seconds with Cypress on a single machine. Sixty route scans at 2.5 s each: wall clock and machines needed 0 50 100 150 sec 150 s serial 1 worker 1 machine 45 s Playwright 4 workers 1 machine 52 s Cypress 60 spec files 4 machines 158 s Cypress any file layout 1 machine Spec-level distribution buys wall clock with machines; test-level distribution buys it with cores.
Spec-level distribution needs one file per scan and one machine per lane, while test-level distribution parallelises a loop as written.

The last axis is determinism under retry, and it cuts both ways. Cypress’s implicit command retry is superb for functional flakiness and slightly dangerous for scanning: cy.checkA11y() is not a retried assertion, so it runs once against whatever DOM exists at that moment, while the cy.get() that preceded it retries and can resolve the instant an element attaches — mid-animation, mid-hydration. Playwright’s auto-waiting locators have exactly the same hazard, but the scan is an explicit await placed after explicit waits, which makes the ordering visible in the diff during review. Test-level retries are the bigger problem in both runners: a scan that fails on attempt one and passes on attempt two is marked flaky and the job goes green, which is precisely the signature of a real intermittent violation such as an error region that occasionally renders after the scan. Playwright’s per-attempt attachments let the two attempts’ results be diffed; Cypress’s per-attempt screenshots carry no rule data unless it was explicitly written out.

Configuration

The scenario that separates the runners in practice is a checkout page with a cross-origin payment iframe and a terms popup. In Playwright, all three documents are scanned by three builders in one test:

// tests/a11y/checkout-surfaces.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

const TAGS = ['wcag2a', 'wcag2aa', 'wcag21aa', 'wcag22aa'];
const blocking = (r: { violations: { impact?: string | null; id: string }[] }) =>
  r.violations.filter((v) => v.impact === 'critical' || v.impact === 'serious');

test('checkout page, payment frame and terms popup', async ({ page, context }) => {
  await page.goto('/checkout');
  await page.getByRole('heading', { name: 'Payment' }).waitFor();

  // 1. The host document, with the vendor frame left in scope on purpose.
  const host = await new AxeBuilder({ page }).withTags(TAGS).analyze();

  // 2. The cross-origin payment frame as its own scan, addressed by frame path.
  const frame = await new AxeBuilder({ page })
    .withTags(TAGS)
    .include(['iframe#psp-card', 'form']) // [frame selector, selector inside it]
    .analyze();

  // 3. The popup arrives as a real Page, so it gets a real builder.
  const [terms] = await Promise.all([
    context.waitForEvent('page'),
    page.getByRole('link', { name: 'Terms of sale' }).click(),
  ]);
  await terms.waitForLoadState('domcontentloaded');
  const popup = await new AxeBuilder({ page: terms }).withTags(TAGS).analyze();

  for (const [label, results] of [['host', host], ['frame', frame], ['popup', popup]] as const) {
    expect(blocking(results).map((v) => `${label}:${v.id}`)).toEqual([]);
  }
});

The Cypress equivalent covers the host document cleanly and needs a workaround for each of the other two. cy.origin() runs a callback in another origin’s context, but the engine must be injected inside that callback because nothing carries across the boundary except serialisable args, and the popup has to be redirected into the same frame:

// cypress/e2e/checkout-surfaces.cy.js
const TAGS = ['wcag2a', 'wcag2aa', 'wcag21aa', 'wcag22aa'];
const OPTS = { runOnly: TAGS, includedImpacts: ['critical', 'serious'] };

describe('checkout accessibility surfaces', () => {
  it('scans the host document', () => {
    cy.visit('/checkout');
    cy.contains('h2', 'Payment').should('be.visible');
    cy.injectAxe();
    // The vendor iframe below is silently out of reach from this context.
    cy.checkA11y(null, OPTS);
  });

  it('scans the terms page by navigating, not by opening a tab', () => {
    // window.open cannot be driven, so force the target into the same frame.
    cy.visit('/checkout', {
      onBeforeLoad(win) {
        cy.stub(win, 'open').callsFake((url) => win.location.assign(url));
      },
    });
    cy.contains('a', 'Terms of sale').click();
    cy.injectAxe(); // a navigation discards the previous injection
    cy.checkA11y(null, OPTS);
  });

  it('scans the payment form on its own origin', () => {
    // Requires experimentalOriginDependencies and the real PSP URL, and it
    // scans that page standalone rather than embedded in the checkout layout.
    cy.origin('https://pay.example.net', { args: { OPTS } }, ({ OPTS }) => {
      cy.visit('/card-form');
      cy.injectAxe();
      cy.checkA11y(null, OPTS);
    });
  });
});

The comments mark the honest difference. Test two scans the terms document in a layout the user never sees. Test three scans the payment form standalone, so it cannot detect the failures that only exist when the form is embedded — a duplicated id colliding with the host page, a contrast failure caused by the host’s inherited background, a heading level that only makes sense in context.

Validation

Prove the reach gap on the application rather than trusting the argument. Count the frames the browser knows about and compare that to what the engine says it tested; the check named frame-tested appears in incomplete for every document the engine could not enter.

// tests/a11y/frame-reach.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test('every frame on checkout was actually scanned', async ({ page }) => {
  await page.goto('/checkout');
  await page.getByRole('heading', { name: 'Payment' }).waitFor();

  const frameCount = page.frames().length; // main frame plus every child
  const results = await new AxeBuilder({ page }).analyze();
  const untested = results.incomplete.filter((v) => v.id === 'frame-tested');

  console.log(`frames=${frameCount} untested=${untested.length}`);
  expect(untested, 'a frame was skipped, so its violations are unknown').toEqual([]);
});

Then diff the rule ids the two runners report for the identical URL. Write both result sets to disk and compare the sorted id lists — the difference is the reach gap expressed as rules rather than as architecture:

# Playwright writes playwright-axe.json; the Cypress task writes cypress-axe.json.
jq -r '.violations[].id' playwright-axe.json | sort -u > pw.ids
jq -r '.violations[].id' cypress-axe.json   | sort -u > cy.ids
diff pw.ids cy.ids
# Typical output on a page with a cross-origin payment frame:
# < color-contrast     # found inside the PSP frame, unreachable from Cypress
# < label              # the PSP card input has no accessible name

Two ids present only on the Playwright side is the expected shape when a cross-origin frame is in play. Zero difference means the page has no unreachable documents and the reach argument does not apply to it.

Recommendation and Migration Cost

For a team with no end-to-end suite yet, choose Playwright and treat @axe-core/playwright as the scanning harness. The reasons are specific to accessibility rather than general preference: cross-origin frames and second tabs are scannable without workarounds, a loop over routes parallelises across cores without becoming sixty files, and the report attachment model puts the rule id, the node and a screenshot one click from a failing test name.

For a team that already runs a healthy Cypress suite, do not migrate. cypress-axe in an existing suite catches the overwhelming majority of what a scanner can catch, and the value of a scan comes from running on every pull request, not from the runner’s badge. Adopt the Cypress a11y testing workflows patterns instead, and if the work is design-system oriented, configuring cypress-axe for component testing is a stronger fit than anything the other runner currently offers, because Cypress component testing is stable and Playwright’s remains experimental.

The middle case is a Cypress suite with a real reach problem — an embedded payment or identity provider, a popup-based consent flow, a scan matrix that has outgrown its machine budget. The cheap move there is not a migration but an addition: a second, small Playwright project that owns accessibility only. In practice that is a playwright.config.ts, one fixture file and one spec file, which is roughly two engineer-days for someone who has not used the runner before, and it leaves the functional suite untouched. A full port of a mature Cypress suite is a different order of magnitude — every custom command becomes a fixture or page-object method, cy.intercept becomes page.route, and cy.origin blocks collapse into ordinary navigation — and it is a project measured in weeks, not something an accessibility requirement should trigger.

Situation, recommendation, migration cost Three rows map starting situations to recommendations: no suite yet leads to Playwright with the axe builder at no migration cost, a healthy Cypress suite leads to cypress-axe with zero migration, and a Cypress suite with frame or tab gaps leads to adding a small Playwright accessibility project for about two engineer-days. What to do, given what already exists starting situation recommendation cost to get there no suite yet Playwright axe builder as harness nothing to migrate healthy Cypress suite no reach problems stay on Cypress add cypress-axe zero, deliberately Cypress with frame or tab gaps add a Playwright a11y project only about 2 engineer-days config, fixture, one spec A full port of a mature suite is weeks of work and is never justified by an accessibility requirement alone.
Only the third row justifies introducing a second runner, and it justifies an accessibility-only project rather than a migration.

Edge Cases and Conditional Guards

  • Component-level scanning inverts the recommendation. Cypress component testing mounts a component into the application frame with a stable API, which makes per-component scans straightforward and fast; Playwright’s component testing is still experimental and its API has changed between minor releases. A design-system team whose primary target is components, not routes, should weigh that heavily.
  • Engine coverage changes results, not just confidence. Playwright ships WebKit and Firefox builds, and color-contrast plus target-size can resolve differently per engine because computed styles and hit-target geometry differ. Cypress runs Chromium-family browsers plus an experimental WebKit build. If a WCAG claim has to hold in Safari, the engine matrix is part of the runner decision.
  • Cross-origin single sign-on. A scan of an authenticated page needs a session. Playwright’s storageState captures cookies and local storage once in a setup project and reuses them, so the identity provider is never visited during the scan. Cypress needs cy.origin() for the provider’s pages, and the engine has to be injected inside each callback — the pattern is worked through in scanning authenticated pages in Cypress a11y runs.

Pipeline Impact

The gate is the runner’s exit code either way, so branch protection does not care which one produced it. What differs is the shape of the job. A Playwright accessibility gate is one job with a shard matrix, and its artifact is a merged HTML report containing per-test attachments; scaling it further is the arithmetic in sharding axe-core scans across parallel CI jobs. A Cypress accessibility gate is one job per parallel lane plus an orchestration dependency, and its artifact is a set of screenshots and videos plus whatever the violation callback wrote to disk.

Budget consequences follow from that. Four Playwright shards of three workers each on standard runners cost four runner-minutes for a sixty-route matrix. Four Cypress lanes cost four runner-minutes plus the orchestration service, and the sixty-spec-file layout that makes the lanes effective also multiplies fixed per-spec startup — roughly two to four seconds each — which is why the measured Cypress number lands above the Playwright one despite using more machines. Retry policy needs an explicit decision in both: set retries to one so infrastructure flakiness does not block a merge, and treat any test marked flaky by the accessibility job as a bug report rather than as noise, because a scan that only sometimes finds a violation has found a real race in the application.

Common Pitfalls

  • Comparing violation counts between the two runners on a page with a cross-origin frame and concluding one engine is stricter — the engine is identical, the reachable DOM is not.
  • Asserting only on violations and never on incomplete, which turns an unscannable frame into a silent pass in both runners.
  • Writing sixty route scans as a loop in one Cypress spec file and then paying for four parallel machines that cannot split it.
  • Stubbing window.open to redirect a popup into the main frame and treating the result as a scan of the popup, when the layout, viewport and inherited styles are all different.
  • Leaving test-level retries on without per-attempt evidence, so an intermittent violation is filed as runner flakiness and never investigated.
  • Migrating a working suite to change runners because of an accessibility requirement, when a second accessibility-only project delivers the same coverage in two days.

FAQ

Do the two runners ever disagree on the same DOM? No. Given an identical document and identical run options, the engine returns identical rule results, because the engine is the same library in both cases. Every disagreement traces to one of three causes: a different reachable document set, different run options (tag lists and includedImpacts filters are easy to drift apart), or a different DOM state because one runner scanned earlier in the render. Reconciling a disagreement always starts by pinning the tag list and printing the incomplete array on both sides.

Is the popup and cross-origin reach difference actually significant, or is it a corner case? It depends entirely on the product. A content site with no embeds loses nothing. A checkout, a banking dashboard, an insurance quote flow or anything using a hosted payment field, an embedded identity provider or a third-party scheduling widget has a substantial fraction of its most legally exposed interface inside a document Cypress cannot enter. Run the frame-reach spec above once against the real application; the answer is a number, not an opinion.

What is the honest downside of choosing Playwright for accessibility work? Component testing is the main one — it is experimental, so a design-system team gets a less stable harness than Cypress offers. Beyond that, the debugging story for a failed scan relies on traces and attachments rather than an interactive time-travelling UI, which some engineers find slower to learn, and there is no built-in orchestration dashboard, so trend reporting has to be assembled from the JSON output rather than read off a hosted service.

Back to Playwright Accessibility Plugin Integration