# Lighthouse CI Baseline Configuration

Lighthouse CI is the accessibility tool teams adopt in an afternoon and stop trusting within a month. The failure is almost never the audit engine — it is the configuration: one run instead of five, a single category threshold instead of per-audit assertions, no stored baseline, and a Chrome launched with defaults that differ from the browser the developer used to reproduce the failure. This guide is part of Web Accessibility Testing Fundamentals & Tool Selection, and it covers how to write a lighthouserc.js that produces the same verdict twice in a row and names the audit that changed.

# Problem Statement

The lhci autorun default path optimises for a first green run, not for a durable gate. Out of the box you get one Lighthouse run per URL, an assertion set that either does not exist or comes from a whole-category preset, and no persistence — so the only thing the job can tell you is a number with no history attached. That produces three distinct classes of failure in a pipeline.

The first is variance. A single Lighthouse run on a shared CI runner is a sample of one from a noisy distribution. Layout-dependent audits such as target-size and color-contrast depend on what actually painted inside the emulated viewport, and a runner that lost its CPU slice to a neighbouring job for 400 ms can paint a different page. A gate that fires on a 0.03 score wobble teaches developers to press re-run, and a gate people re-run is not a gate.

The second is resolution. categories:accessibility collapses roughly sixty weighted audits into one number between zero and one. When that number drops from 0.95 to 0.91, the job output tells you the category regressed; it does not tell you that a form label disappeared. Worse, audits that do not apply to a page are dropped from the weighted average entirely, so the same score means different things on / and /checkout and cannot be compared between them.

The third is the missing baseline. Accessibility scores are only interesting as a delta against the branch you are merging into. Without an upload target, every build is an island: you know the score is 0.91, you do not know whether main is 0.91 too. Fixing all three is a configuration job, and the config file has exactly three blocks to fix.

Key implementation targets:

• A lighthouserc.js (not .json) whose collect, assert and upload blocks are each commented and each independently runnable through lhci collect, lhci assert and lhci upload.
• numberOfRuns set to 5 with aggregationMethod: 'median' declared explicitly on every assertion, so the verdict comes from the middle of the distribution rather than the most favourable run.
• Assertions written against individual accessibility audit IDs at minScore: 1, with the category score kept as a non-blocking warn trend line.
• A collect block that either serves a built staticDistDir or starts the real application server behind a startServerReadyPattern, plus explicit chromeFlags for a containerised runner.
• A baseline that survives the job: temporary-public-storage for a first pass, a self-hosted LHCI server once you need to diff against an ancestor commit.
• A CI invocation that keeps the full .lighthouseci directory as an artifact so a failing assertion can be re-read without re-running the browser.

# Prerequisites

• Node.js 20.x or newer — @lhci/cli 0.14 drops Node 16 and warns on Node 18.
• A Chrome or Chromium binary on the runner, or CHROME_PATH pointing at one (npx playwright install --with-deps chromium is the least surprising way to get it).
• A production build command that emits a static directory, or a start command that logs a predictable ready line.
• Write access to the repository’s CI secrets if you plan to use a self-hosted LHCI server (LHCI_TOKEN, LHCI_SERVER_BASE_URL).
• actions/checkout configured with fetch-depth: 0, because ancestor-commit resolution needs real git history rather than a shallow clone.
• A shortlist of four to six URLs that represent distinct templates, not four instances of the same template.
• Agreement on which audits are allowed to block a merge — write it down before the first error assertion lands.

# 1. Install and Initialise

Install the CLI as a dev dependency rather than invoking it through npx from the registry each time; a floating version changes the bundled Lighthouse, and a changed Lighthouse changes audit IDs and weights without a commit to blame.

npm install --save-dev @lhci/cli@0.14.x   # pinned minor: Lighthouse is bundled inside
npx lhci --version                        # prints the CLI and the Lighthouse it wraps
npx lhci healthcheck --fatal              # non-zero exit if Chrome or git context is missing

lhci wizard exists and is worth running once, but only to create a project on a server and print its build token — it writes a minimal lighthouserc.json that you will replace immediately. Use the JavaScript form of the config instead. lighthouserc.js is a CommonJS module, which means it can carry comments explaining why a threshold is what it is, and it can read an environment variable to switch behaviour between a pull-request run and a main run without maintaining two files.

// lighthouserc.js — read by every lhci subcommand from the repository root.
const isMain = process.env.GITHUB_REF === 'refs/heads/main';

module.exports = {
  ci: {
    collect: {
      // 5 samples on a PR is the smallest number that gives a stable median
      // for layout-dependent audits; main can afford more.
      numberOfRuns: isMain ? 7 : 5,
    },
    assert: {},
    upload: {},
  },
};

Keep the file at the repository root. Every subcommand resolves it from the current working directory upwards, so a config that lives in a ci/ folder needs --config=ci/lighthouserc.js on every invocation and someone will eventually forget it.

# 2. The collect Block: URLs, Servers and Chrome Flags

collect decides what gets audited and under what conditions, and it has two mutually exclusive modes. With staticDistDir, Lighthouse CI starts its own static file server against a built directory and resolves relative paths against it; the url array then holds paths like /pricing/. With startServerCommand, you own the server and the url array holds absolute origins. Static mode is faster and immune to a flaky application boot, but it cannot exercise server-side rendering, API-backed pages, redirects or authenticated routes — which for most applications means it cannot audit the pages where the accessibility bugs actually live.

The settings object is passed straight through to Lighthouse, which is where onlyCategories and chromeFlags belong. Restricting to the accessibility category cuts a five-run collection from minutes to under a minute because none of the performance tracing runs, and it removes an entire category of score noise from the report. Chrome flags matter in containers: without --no-sandbox the browser refuses to start as root, and without --disable-dev-shm-usage it crashes partway through on the 64 MB /dev/shm that Docker gives a container by default. Both symptoms surface as a Lighthouse “protocol timeout”, which reads like a network problem and is not one.

// lighthouserc.js — the collect block, server mode.
module.exports = {
  ci: {
    collect: {
      numberOfRuns: 5,
      startServerCommand: 'npm run preview -- --port 4173 --host 127.0.0.1',
      // Anchored regex: matching a bare "4173" also matches a webpack hash.
      startServerReadyPattern: 'Local:\\s+http://127\\.0\\.0\\.1:4173',
      startServerReadyTimeout: 60000, // ms; a cold Vite preview on a shared runner
      url: [
        'http://127.0.0.1:4173/',
        'http://127.0.0.1:4173/pricing/',
        'http://127.0.0.1:4173/checkout/cart/',
        'http://127.0.0.1:4173/support/contact/',
      ],
      settings: {
        onlyCategories: ['accessibility'], // drops perf tracing: ~4x faster
        // Emulation must be identical on every runner or target-size drifts.
        formFactor: 'mobile',
        screenEmulation: {
          mobile: true, width: 412, height: 823,
          deviceScaleFactor: 1.75, disabled: false,
        },
        throttlingMethod: 'simulate',
        maxWaitForLoad: 30000, // ms; hydration on a cold cache
        chromeFlags: [
          '--no-sandbox',              // required when the container runs as root
          '--disable-dev-shm-usage',   // Docker's 64MB /dev/shm crashes renderers
          '--disable-gpu',
          '--headless=new',
          '--force-prefers-reduced-motion', // stops CSS animation mid-audit
        ],
      },
    },
  },
};

Pick the URL list for template coverage, not for traffic. Four routes that render four different layouts find more than twenty product pages built from one template, and a five-run collection over four URLs is twenty Lighthouse runs — enough that adding routes carelessly turns a two-minute job into an eight-minute one. If the routes you care about sit behind a login, drive the browser yourself with a puppeteerScript or hand the authenticated pages to a scanner that carries session state, as described in axe-core configuration and setup.

# 3. Accessibility Assertions: Per Audit, Not Per Category

This is the block that decides whether the gate is useful. Lighthouse CI supports three presets — lighthouse:all, which requires a perfect score on every audit, lighthouse:recommended, which is the same set with performance metrics and several known-noisy audits downgraded to warn, and lighthouse:no-pwa. All three are whole-report presets: they assert against performance, SEO and best-practices audits as well, so dropping one into an accessibility-only pipeline produces assertion results for audits that were never collected. Start from lighthouse:recommended only if you genuinely gate the whole report. For an accessibility gate, list the assertions explicitly — anything you do not list is simply not asserted, which is exactly the behaviour you want.

Aggregation is the detail that silently breaks multi-run configurations. Setting numberOfRuns: 5 does not by itself mean the assertion sees the median: Lighthouse CI’s default aggregation is optimistic, meaning it evaluates the run most likely to satisfy the assertion. On a page whose median score is 0.88 and whose best run is 0.96, an optimistic assertion at minScore: 0.90 passes. Declare aggregationMethod: 'median' on every assertion — or 'median-run' if you want all values taken from the single run that was median overall, which keeps the numbers in a report internally consistent.

The assertion set below is the shape to copy. Individual audits that are deterministic — a missing alt, an unlabelled control, an invalid ARIA attribute value — sit at error with minScore: 1, because these audits are binary and a score of anything other than 1 means at least one node failed. Audits whose result depends on layout or on judgement sit at warn. The category score sits at warn with a deliberately loose floor, where it functions as a trend line rather than a gate; which audit IDs deserve which level is worked through criterion by criterion in setting up Lighthouse CI thresholds for WCAG 2.2 AA.

// lighthouserc.js — the assert block. No preset: nothing is asserted implicitly.
const median = { aggregationMethod: 'median', minScore: 1 };

module.exports = {
  ci: {
    assert: {
      // Report passing assertions too, so the log proves an audit ran at all.
      includePassedAssertions: true,
      assertions: {
        // Binary audits: any score below 1 means a real failing node exists.
        'image-alt': ['error', median],
        'input-image-alt': ['error', median],
        'label': ['error', median],
        'button-name': ['error', median],
        'link-name': ['error', median],
        'select-name': ['error', median],
        'aria-required-attr': ['error', median],
        'aria-valid-attr-value': ['error', median],
        'aria-hidden-focus': ['error', median],
        'document-title': ['error', median],
        'html-has-lang': ['error', median],
        'color-contrast': ['error', median],
        // Layout- or judgement-dependent: informative, never merge-blocking.
        'target-size': ['warn', median],
        'heading-order': ['warn', median],
        'bypass': ['warn', median],
        'link-in-text-block': ['warn', median],
        // The category score is a trend line, not a gate. 0.85 catches a
        // collapse; it deliberately does not catch a single regression.
        'categories:accessibility': [
          'warn',
          { aggregationMethod: 'median', minScore: 0.85 },
        ],
      },
    },
  },
};

Two behaviours of this block are worth internalising. assert reads the JSON that collect already wrote into .lighthouseci/, so you can iterate on thresholds in seconds with npx lhci assert and no browser. And an audit that is not applicable to a page — video-caption on a page with no video — is reported as not applicable rather than as a pass, and is excluded from the category’s weighted average. That exclusion is the reason category scores are not comparable between URLs, and the reason per-audit assertions are the only assertions that mean the same thing everywhere.

# 4. Baseline Storage and Comparison

A score without a baseline is trivia. Lighthouse CI’s upload block gives three targets, and the choice is a trade between setup cost and the ability to answer “is this worse than main?”.

temporary-public-storage uploads the report to a Google-hosted bucket and prints a URL into the job log. It needs no secrets and no infrastructure, the link is public to anyone who has it, and reports expire after a few days. It is the right first step and the wrong permanent answer: nothing is stored server-side that a later build can compare against, so it gives you a shareable report and no baseline.

filesystem writes the reports into a directory you nominate, which is the right target when you already have an artifact store and a dashboard — pair it with the trend tooling in tracking accessibility violation trends across sprints rather than building a second one.

The lhci target points at a self-hosted Lighthouse CI server, and it is the only option that produces a real baseline. The server stores every build against its commit hash and branch, and on a pull-request build it resolves the ancestor commit — the merge base with the base branch — then renders the diff per URL and per audit. This is why the checkout step needs full git history: with a shallow clone the ancestor hash cannot be computed, the server has nothing to compare against, and the build silently shows up as a first-of-its-kind rather than as a regression.

Running the server is a container and a database. It stores builds, exposes a web UI with per-audit history for every URL, and issues one build token per project.

# One-off: create the project and print its build token.
npx lhci server --port 9001 --storage.storageMethod=sql \
  --storage.sqlDialect=postgres \
  --storage.sqlConnectionUrl="$LHCI_DB_URL" &
npx lhci wizard   # choose "new-project", answer the prompts, copy the build token

# In CI: the token authorises writes for this project only.
export LHCI_SERVER_BASE_URL="https://lhci.internal.example"
export LHCI_TOKEN="$LHCI_BUILD_TOKEN"
npx lhci upload --target=lhci   # reads .lighthouseci/ written by collect

The matching config block keeps the target and the URL-normalisation rules together. urlReplacementPatterns is not cosmetic: without it, a preview deployment on a per-branch hostname is treated as a different URL from the one main audited, and the server has no history to diff.

// lighthouserc.js — the upload block.
module.exports = {
  ci: {
    upload: {
      target: 'lhci',
      serverBaseUrl: process.env.LHCI_SERVER_BASE_URL,
      token: process.env.LHCI_TOKEN,
      // Collapse per-branch preview hosts onto one canonical origin so that
      // the server compares like with like across builds.
      urlReplacementPatterns: [
        's#^https://[^/]+#https://app.example#',
        's#\\?.*##', // drop query strings: ?utm= would fork the history
      ],
      // Posts a commit status per URL when the app is installed on the repo.
      githubAppToken: process.env.LHCI_GITHUB_APP_TOKEN,
    },
  },
};

# 5. The CI Invocation and Its Artifacts

lhci autorun runs collect, assert and upload in sequence and exits non-zero if any error assertion failed. That single-command form is fine, but it has one property worth overriding: a failed assertion normally stops the pipeline before the upload happens, which means the run you most want to look at is the one that never got stored. Run the subcommands separately and let upload happen regardless.

name: lighthouse-a11y-baseline
on:
  pull_request:
    paths:
      - 'src/**'
      - 'public/**'
      - 'lighthouserc.js'
      - '.github/workflows/lighthouse-a11y-baseline.yml'
concurrency:
  group: lhci-a11y-${{ github.head_ref }}
  cancel-in-progress: true
jobs:
  lhci-accessibility:
    runs-on: ubuntu-24.04
    timeout-minutes: 25
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # ancestorHash resolution needs real history
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npm run build
      - name: Collect five runs per route
        run: npx lhci collect   # writes .lighthouseci/lhr-*.json
      - name: Assert the accessibility audits
        id: assert
        continue-on-error: true   # let the upload run even on a red assertion
        run: npx lhci assert
      - name: Upload to the LHCI server
        if: always()
        run: npx lhci upload --target=lhci
        env:
          LHCI_SERVER_BASE_URL: ${{ secrets.LHCI_SERVER_BASE_URL }}
          LHCI_TOKEN: ${{ secrets.LHCI_TOKEN }}
          LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: lighthouseci-reports
          path: .lighthouseci/   # lhr JSON + HTML for every run
          retention-days: 14
      - name: Fail the job if assertions failed
        if: steps.assert.outcome == 'failure'
        run: exit 1   # restores the blocking behaviour autorun would have had

The final step is the one people delete by accident. continue-on-error on the assert step makes the step green in the UI while recording its real outcome in steps.assert.outcome, so the explicit exit 1 is what actually blocks the merge. Remove it and you have a permanently advisory job.

# Pipeline Integration

Lighthouse CI’s exit code is the whole integration surface: zero when every error assertion passed, non-zero otherwise, with warn assertions printed and ignored. That maps cleanly onto branch protection — make the job a required status check and the error list becomes the merge contract. Keep the list short enough that a failure is always a real bug, because the cost of one false block is roughly ten legitimate blocks’ worth of credibility.

The assertion output is machine-readable enough to annotate a pull request without extra tooling. lhci assert prints one line per failed assertion with the audit ID, the URL, the expected value and the actual value; piping it into the job summary gives reviewers the audit name without opening an artifact. If the LHCI GitHub App is installed, the upload step additionally posts one commit status per audited URL linking to the stored report, which is the fastest path from a red check to the failing element.

Two staging patterns are worth adopting. First, run the accessibility-only collection on every pull request and the full-category collection nightly on main; the accessibility subset is fast enough for the critical path and the full report is not. Second, introduce every new error assertion as a warn first and promote it after it has been green for a sprint — the soak-then-promote mechanics are covered in auto-fail versus warning workflows. If the runner is a container, the caching and binary-path details are in running Lighthouse CI in a Docker-based pipeline.

One integration decision needs stating explicitly, because getting it wrong wastes months: Lighthouse CI is not a replacement for a rule-level scanner on the blocking path. Its accessibility audits are a subset of axe-core’s rule set, evaluated once per audit rather than once per node, under mobile emulation. When the two tools disagree — and they will — Lighthouse accessibility score versus axe violation counts shows how to join the two result sets on rule ID and read the difference; the tool-choice argument for the gate itself is made in axe-core vs Lighthouse CI for PR gating.

# Troubleshooting and Flaky-Test Mitigation

Chrome fails to start, or the run dies with a protocol timeout. In a container this is almost always the sandbox or shared memory, not the network. Add --no-sandbox and --disable-dev-shm-usage to chromeFlags, or start the container with --shm-size=1g. Confirm the binary first: npx lhci healthcheck --fatal reports a missing Chrome, a missing git context and an unreadable config in one pass, and it costs two seconds.

The first URL fails and the rest pass. The ready pattern matched something that was not the server. startServerReadyPattern is a regex tested against the command’s stdout, and a loose pattern like 4173 matches a bundle hash printed during startup, so Lighthouse navigates before the port is listening. Anchor the pattern on text only the listening server prints, and raise startServerReadyTimeout for a cold build.

target-size fails intermittently on the same commit. This audit measures rendered geometry, so it is sensitive to fonts and to animation. A web font that loads after first paint reflows a button below 24 by 24 CSS pixels for a few frames; a CSS transition caught mid-flight reports the transient size. Add --force-prefers-reduced-motion to the Chrome flags, keep the audit at warn, and treat the median over five runs as the number.

Scores drop by a few points with no application change. Compare the Lighthouse version between the two runs. @lhci/cli bundles Lighthouse, so an unpinned install can change the audit set, the weights and the axe-core version inside it — three ways to move a score with no commit responsible. Pin the CLI to a minor version and treat a Lighthouse upgrade as its own pull request with an expected baseline shift.

Hydration-dependent violations appear and disappear. Lighthouse snapshots the DOM at the end of the page-load lifecycle, which for a heavy client-rendered application can precede the render of the component that actually fails. Raise maxWaitForLoad, and if a route only settles after a user action, drive it with a puppeteerScript before the audit rather than hoping the timing holds.

The server shows every build as a new baseline. The ancestor hash is unresolved. Set fetch-depth: 0 on the checkout, verify that the base branch ref exists locally, and check that urlReplacementPatterns collapses the preview hostname — a branch-specific origin makes every URL unique and therefore historyless.

# Common Pitfalls

• Leaving numberOfRuns at 1, then attributing the resulting score wobble to the audit engine rather than to a sample of one on a shared runner.
• Setting numberOfRuns: 5 but never setting aggregationMethod, so the assertion silently evaluates the friendliest of the five runs.
• Gating on categories:accessibility alone, which cannot distinguish “a label went missing” from “this page has fewer applicable audits than the last one”.
• Using lighthouse:recommended together with onlyCategories: ['accessibility'], which asserts performance and SEO audits that were never collected.
• Running lhci autorun and losing the upload whenever assertions fail, so the report for the interesting build is the one that was never stored.
• Installing @lhci/cli with a floating range, letting a bundled Lighthouse upgrade move every threshold at once.
• Auditing four URLs that all render the same template, producing four copies of one finding and zero coverage of the checkout flow.
• Forgetting fetch-depth: 0, which turns a baseline-comparison server into an expensive place to store unrelated snapshots.
• Comparing scores between two URLs, when not-applicable audits mean the two scores have different denominators.

# FAQ

Should the accessibility category score ever be a blocking assertion? Only as a floor that catches catastrophes, not as the gate. A weighted average over roughly sixty audits moves by a couple of points when a single audit flips, so a threshold tight enough to catch one regression will also fire on emulation noise, and a threshold loose enough to be stable will miss real bugs. Assert the individual audit IDs at minScore: 1 for blocking, and keep the category at warn with a floor around 0.85 so a genuine collapse still shows up in the log.

How many runs are actually needed, and what does each one cost? Five is the practical minimum for a stable median on layout-sensitive audits, and seven is worth it on main where nothing is waiting on the result. With onlyCategories: ['accessibility'] a run against a locally served page takes a few seconds rather than the twenty-plus seconds a full-category run takes, so five runs over four URLs fits comfortably inside a pull-request job. If the job is too slow, cut URLs before cutting runs — fewer samples buys flakiness, and flakiness costs more time than it saves.

Can Lighthouse CI audit pages behind authentication? Yes, through puppeteerScript, which receives the browser and the page before each audit and can log in, set a cookie or dismiss a consent banner. The caveat is that the script runs for every run of every URL, so a slow login multiplies by numberOfRuns; seed the session with a cookie or a pre-authenticated storage state instead of driving the login form when you can. For flows that need several steps of interaction before the interesting DOM exists, a Playwright-driven scanner is a better fit than Lighthouse.

What is the difference between median and median-run aggregation? median takes the median of the values for that specific assertion across runs, so two assertions can end up sourced from two different runs. median-run first picks the run that was median overall and then reads every value from that one run, which keeps the numbers self-consistent — useful when a human is going to read the report and ask why the score and the audit list do not add up. For a pass/fail gate on binary audits the two behave the same; for mixed reporting, prefer median-run.

Why does an audit show as not applicable instead of passing? Lighthouse marks an audit not applicable when the page contains nothing for it to check — video-caption with no video element, td-has-header with no data tables — and excludes it from the category’s weighted average entirely. That is correct behaviour, but it means the denominator of the score is page-specific, so scores are only comparable to earlier scores for the same URL. It is also the quiet reason a per-audit assertion can pass on a page that never exercised the rule at all; use includePassedAssertions: true to see which audits actually ran.

# Related

• Setting Up Lighthouse CI Thresholds for WCAG 2.2 AA — which audit IDs map to which success criteria, and which belong on error.
• Lighthouse Score vs axe Violation Counts — join both result sets on rule ID and read the disagreement.
• Web Accessibility Testing Fundamentals & Tool Selection — where a score-based budget sits among the five scanning engines.
• Lighthouse CI in a Docker Pipeline — container images, Chrome binaries and cache layers for this job.
• Reporting, Dashboards & Violation Tracking — turning stored builds into a trend the team actually looks at.