Measured on the real web, not on fixtures.
From 2026-09-22 to 2026-09-23 we put the scanner through a controlled audit: 58 real websites, 5 pages each, scanned through production before and after a round of accuracy fixes. Then we went back to every finding the fixes removed and checked it on the live page. This is what we found, including the parts that did not flatter us.
Findings
860
from 823 · +4%
Duplicate rows
0
from 14 · the same defect, reported once
Findings with no rule id
0
from 0 · every row is filterable
September 2026 field audit IV · 51 of 58 sites comparable (3 blocked the scanner in both runs, 4 changed reachability between runs) · builds 5364104 → 2d89faa · ran 2026-09-22 to 2026-09-23
Engine by engine
Rule-class counts are a poor scoreboard for an accuracy fix, because a fix that gives a row an id it previously lacked moves that row between classes without removing anything. Engine totals cannot be moved by relabelling.
| Source | Before | After | Change |
|---|---|---|---|
| arc-style | 79 | 83 | +4 |
| axe-core | 177 | 185 | +8 |
| ibm-equal-access | 199 | 165 | −34 |
| lighthouse | 19 | 16 | −3 |
| pa11y | 127 | 111 | −16 |
| In-house analyzers, reported through the same field | |||
| focus-visibility | 0 | 78 | +78 |
| keyboard-nav-tester | 3 | 1 | −2 |
| language-check | 1 | 0 | −1 |
| mobile-accessibility | 218 | 197 | −21 |
| reflow-320 | 0 | 24 | +24 |
Did the fixes remove anything real?
Fewer findings proves nothing on its own. Every occurrence present in the first run and absent from the second - 2,583 of them - was re-measured on the live page with a method independent of the scanner: a verifier that shares no code with the scanner. Names, roles and hidden-ness come from Chromium's own accessibility tree over the DevTools protocol; contrast from rendered pixels, the glyphs isolated by screenshotting the text and then the same box with the text made transparent; id references by token lookup in the element's own root; focusability by calling focus() and reading the active element. Each occurrence keeps the scanner's page preparation (1920x1080, light theme, reduced motion, the scroll primer). The before run is the field sweep of 22-23 September on 5364104; the after run rescanned every sweep site on 2d89faa, the build with PRs #47 to #50. Every occurrence the comparison counted has a row here: target-size occurrences, which the verifier hands to a separate mobile pass, are counted as indeterminate rather than dropped. The bias runs toward reporting a lost true positive, not toward excusing the removal.
| Correctly removed | 87 |
| Still counted, just not listed | 37 |
| Same element, listed under another row | 375 |
| Re-bucketed under another rule | 4 |
| Genuinely lost | 30 |
| Removed by a documented policy | 16 |
| Indeterminate (dynamic pages)includes 18 not explained | 2,034 |
| Justified - the verifier confirms the failure on the live page | 523 |
| Borderline - at exactly the 24px threshold | 0 |
| Manufactured by the fixes | 22 |
| Indeterminate (dynamic pages) | 1,219 |
The 30 genuinely lost occurrences are on ten sites: syntax-highlighted code tokens on ionicframework.com (8) and angular.dev's editor (3) and a primary button on shoelace.style (3), all contrast; five labelledby references and one contrast row on vimeo.com; the skip link and main landmark of one detik.com page (3); two labelledby references on gymshark.com; two unnamed icons on jotform.com; an OneTrust reference and a heading level on khanacademy.org; a heading level on naver.com; and a language button on yle.fi whose visible text is not in its name. Each is filed as an open defect against the after-build. The 18 unexplained rows are occurrences the reconciliation could not resolve on the live page hours later and are shown inside Indeterminate; they are not evidence either way. The indeterminate rows are pages the other run did not analyse, target-size occurrences (judged by a separate mobile pass this census does not fold in), dynamic content whose selectors no longer resolve, and rows inside consent or ad vendors' frames.
Removed by a documented policy, 16 occurrences. 16 occurrences were removed by decisions the field sweep verified by hand, where this verifier keeps the literal reading: a tabindex on a scrolling code block needs no widget role (docusaurus.io, ionicframework.com, 6); an empty, zero-height listbox is not a nameless control (canada.ca, 2); inactive pager links are exempt from 1.4.3 (hurriyet.com.tr, 2); text hidden by image replacement is never painted (gov.uk); an icon-font ligature is not a visible label (angular.dev); an icon inside a named role=img needs no name of its own (shoelace.style); and a card link named by the call to action it shows meets 2.5.3 (vimeo.com, 3). They are neither counted as correctly removed nor as lost.
Where the removals came from
Only the classes the fixes touched. Everything else moved with the web, not with the scanner, and is not attributed.
| Rule class | Before | After | What changed |
|---|---|---|---|
| mobile-target-unreachable-advisory | 46 | 30 | opacity, axis-aware clipping, whole scroller chain |
| ibm-fieldset_label_valid | 7 | 0 | named fieldsets and single-field wrappers |
| ibm-aria_attribute_valid | 13 | 7 | prohibited ARIA that browsers ignore at no cost; aria-checked on native inputs |
| ibm-label_content_exists | 7 | 2 | label text, title or a named control |
| link-name | 8 | 3 | icon-font glyphs are not names; accessibility-tree fallback |
| mobile-target-size-aa-fail | 11 | 6 | inline exception; spacing ignores fixed layers and other scroll boxes |
| ibm-text_contrast_sufficient | 29 | 24 | backgrounds not behind the text; fixed overlays |
| ibm-aria_eventhandler_role_valid | 5 | 2 | redundant mouse targets, empty handlers, shadow-rendered buttons |
| input-label-missing | 3 | 1 | an empty label named by its title; names from the accessibility tree |
| ibm-svg_graphics_labelled | 27 | 25 | empty, unrendered or img-hosted svgs |
| ibm-aria_hidden_nontabbable | 8 | 6 | hidden content nothing inside can focus |
| ibm-element_tabbable_role_valid | 21 | 19 | tabbable scroll containers |
| keyboard-trap | 3 | 1 | closed shadow roots and documented Escape exits |
| ibm-aria_role_valid | 5 | 4 | role=none on li is permitted |
| label-title-only | 3 | 2 | judged by the name's real source |
| label-content-name-mismatch | 12 | 11 | icon-font text; a card named by its call to action |
| blink | 1 | 0 | an empty blink has nothing to blink |
| aria-role-invalid | 3 | 3 | role="text" is a low note, not an invalid role |
| ibm-aria_role_allowed | 3 | 3 | role="text" is a low note |
| aria-prohibited-attr | 1 | 1 | prohibited ARIA that browsers ignore at no cost |
| ibm-aria_descendant_valid | 2 | 2 | an image flattened inside a named button is a note |
| ibm-input_label_exists | 3 | 3 | collapsed listbox; names from the accessibility tree |
| aria-hidden-focus | 5 | 5 | hidden content nothing inside can focus |
| page-has-heading-one | 10 | 10 | checked against the live page, shadow roots included |
| ibm-label_name_visible | 7 | 7 | a card named by its call to action |
| target-size | 17 | 17 | 2.5.8 inline and spacing exceptions |
| link-in-text-block | 4 | 4 | a link 3:1 from its sentence |
| color-contrast | 16 | 16 | CSS filters, image-replaced text, inactive components, brand names |
| tabindex | 1 | 1 | best-practice rules cap at medium |
| button-name | 7 | 8 | names read from Chromium's accessibility tree |
| duplicate-id | 9 | 10 | labelledby copies that render the same text |
| reflow-horizontal-scroll | 0 | 24 | new: 1.4.10 measured at 320 CSS px on every page |
| focus-indicator-invisible | 0 | 78 | new: 2.4.7 measured in pixels on the first two pages |
Every site
Scores moved on 38 of 51 sites: 22 up, 16 down. A score can fall because the report got more accurate - when two engines' reports of one defect merge into a single row, the row keeps the higher severity.
| Site | Findings before | After | Score |
|---|---|---|---|
| apnews.com | 27 | 17 | 51 → 56 |
| gov.hk | 10 | 4 | 84 → 89 |
| mozilla.org | 12 | 7 | 77 → 79 |
| techcrunch.com | 28 | 23 | 54 → 55 |
| yle.fi | 19 | 14 | 80 → 76 |
| khanacademy.org | 11 | 7 | 76 → 89 |
| ionicframework.com | 31 | 28 | 51 → 52 |
| svt.se | 7 | 4 | 89 → 100 |
| ikea.com | 12 | 10 | 84 → 83 |
| npr.org | 29 | 27 | 61 → 65 |
| a11yproject.com | 7 | 6 | 87 → 89 |
| airbnb.com | 17 | 16 | 65 → 65 |
| canada.ca | 7 | 6 | 89 → 100 |
| gov.uk | 4 | 3 | 89 → 100 |
| developer.mozilla.org | 12 | 11 | 83 → 83 |
| shoelace.style | 16 | 15 | 70 → 72 |
| svelte.dev | 11 | 10 | 79 → 89 |
| vuejs.org | 17 | 16 | 59 → 59 |
| deque.com | 7 | 7 | 89 → 89 |
| docusaurus.io | 14 | 14 | 79 → 75 |
| design-system.service.gov.uk | 3 | 3 | 100 → 100 |
| naver.com | 29 | 29 | 59 → 59 |
| nextjs.org | 27 | 27 | 74 → 70 |
| rust-lang.org | 6 | 6 | 87 → 87 |
| squarespace.com | 8 | 8 | 61 → 80 |
| typeform.com | 18 | 18 | 75 → 67 |
| webaim.org | 5 | 5 | 87 → 88 |
| allbirds.com | 15 | 16 | 61 → 64 |
| bbc.com/hindi | 7 | 8 | 53 → 89 |
| bbc.com/persian | 6 | 7 | 59 → 89 |
| monzo.com | 16 | 17 | 75 → 78 |
| notion.com | 17 | 18 | 64 → 53 |
| rnib.org.uk | 4 | 5 | 77 → 79 |
| ted.com | 25 | 26 | 67 → 53 |
| wordpress.org | 5 | 6 | 100 → 89 |
| astro.build | 6 | 8 | 89 → 87 |
| duolingo.com | 22 | 24 | 49 → 49 |
| go.dev | 17 | 19 | 60 → 55 |
| gymshark.com | 38 | 40 | 50 → 50 |
| mit.edu | 12 | 14 | 78 → 78 |
| angular.dev | 24 | 27 | 68 → 60 |
| theguardian.com | 23 | 27 | 54 → 63 |
| shopify.com | 13 | 18 | 79 → 76 |
| whitehouse.gov | 11 | 16 | 87 → 83 |
| g1.globo.com | 24 | 30 | 53 → 54 |
| detik.com | 43 | 50 | 50 → 44 |
| service-public.fr | 1 | 8 | 100 → 79 |
| hurriyet.com.tr | 25 | 33 | 51 → 51 |
| gov.br | 19 | 28 | 70 → 67 |
| jotform.com | 24 | 33 | 49 → 49 |
| vimeo.com | 32 | 41 | 49 → 49 |
What this page does not claim
- Not "4% more real defects found." Findings rose by that much. A total moving in either direction mixes removed false positives, checks that were switched on between the runs and changes on the sites themselves; the census above is the only honest breakdown we have.
- Not a false-positive rate. Producing one requires re-verifying every finding that remains, on the live page. We re-verified what changed. A single percentage would be more quotable and less true.
- Not deterministic. Live sites ship new markup, rotate carousels and block scanners between runs. 3 sites were unreachable in both runs and are excluded; a site that changed reachability between runs is excluded too.
- Not the whole story. 84 defects were found and fixed in this audit, each with a regression test proven to fail without it. Some were in the fixes themselves, found by the census. The open items are listed in the repository.
Reproduce it
- Every number on this page is computed from the audit's raw result
files by
scripts/build-proof.pyand committed with the site; none is typed by hand. - The fixture benchmark is the other half of the evidence: precision and recall on hand-annotated ground truth.
- The scanner audited here runs on every pull request through the accessibility GitHub Action.
- 5,126 backend tests pass in both CI modes on the build that shipped these fixes.
- The raw figures for this edition: 2026-09-23.json. Other editions are listed in the field audit series.