Measured on the real web, not on fixtures.
From 2026-09-26 to 2026-09-27 we put the scanner through a controlled audit: 56 real websites, 5 pages each, scanned through production before and after a round of accuracy fixes. Then we went back to every finding the fixes removed and checked it on the live page. This is what we found, including the parts that did not flatter us.
Findings
1,263
from 1,438 · −12%
Duplicate rows
0
from 32 · the same defect, reported once
Findings with no rule id
0
from 0 · every row is filterable
September 2026 field audit V · 53 of 56 sites comparable (3 blocked the scanner in both runs, 0 changed reachability between runs) · builds 3e26cc9 → 0bc595c · ran 2026-09-26 to 2026-09-27
Engine by engine
Rule-class counts are a poor scoreboard for an accuracy fix, because a fix that gives a row an id it previously lacked moves that row between classes without removing anything. Engine totals cannot be moved by relabelling.
| Source | Before | After | Change |
|---|---|---|---|
| arc-style | 141 | 139 | −2 |
| axe-core | 330 | 335 | +5 |
| ibm-equal-access | 226 | 201 | −25 |
| lighthouse | 34 | 30 | −4 |
| pa11y | 202 | 191 | −11 |
| In-house analyzers, reported through the same field | |||
| focus-visibility | 197 | 48 | −149 |
| keyboard-nav-tester | 1 | 5 | +4 |
| label-in-name | 16 | 15 | −1 |
| language-check | 6 | 5 | −1 |
| mobile-accessibility | 277 | 276 | −1 |
| reflow-320 | 8 | 18 | +10 |
Did the fixes remove anything real?
Fewer findings proves nothing on its own. Every occurrence present in the first run and absent from the second - 3,741 of them - was re-measured on the live page with a method independent of the scanner: a verifier that shares no code with the scanner. Names, roles and hidden-ness come from Chromium's own accessibility tree over the DevTools protocol, a name made only of an icon font's private-use glyphs counts as no name, and a valid aria-level sets a heading's level; contrast from rendered pixels, the glyphs isolated by screenshotting the text and then the same box with the text made transparent; id references by token lookup in the element's own root; focusability by calling focus() and reading the active element. Each occurrence keeps the scanner's page preparation (1920x1080, light theme, reduced motion, the scroll primer), and Arabic consent banners are declined in Arabic. A page that answers the verifier with an HTTP error, as a bot challenge does, is not judged. The before run is the Arabic field sweep of 26 September on 3e26cc9 and the after run rescanned every sweep site on 27 September on 0bc595c, the build with PRs #88 to #90; both through production, five pages per site, from the same account. Two sites that failed once in the after run (saudia.com timed out, uaeu.ac.ae did not load) were scanned again, as the before run did with its own failures. Every occurrence the comparison counted has a row here: target-size occurrences, which the verifier hands to a separate mobile pass, are counted as indeterminate rather than dropped. The bias runs toward reporting a lost true positive, not toward excusing the removal.
| Correctly removed | 61 |
| Still counted, just not listed | 102 |
| Same element, listed under another row | 388 |
| Re-bucketed under another rule | 7 |
| Genuinely lost | 19 |
| Indeterminate (dynamic pages)includes 38 not explained | 3,164 |
| Justified - the verifier confirms the failure on the live page | 562 |
| Borderline - at exactly the 24px threshold | 0 |
| Manufactured by the fixes | 40 |
| Indeterminate (dynamic pages) | 1,847 |
The 19 lost occurrences are on eight sites. Sixteen are the pages, not the scanner. Eight aria-hidden rows on hukoomi.gov.qa were measured on a maintenance page in the before run; on the live page the content is aria-hidden only while its cookie dialog is open, and that dialog keeps keyboard focus inside itself, so Tab never reaches the hidden content. Four heading-order rows on talabat.com are rotating content: the row counts the same seven occurrences on the same two pages in both runs, and the headings it lists rotate between loads. On careem.com and zatca.gov.sa the same rule still reports the same page, with as many occurrences or more (3 and 3; 5 and 6), and lists other carousel items. Two rows on echoroukonline.com are its privacy-settings button, which the after run still reports on the site's four other pages. Three are open items, filed against the after-build: two contrast rows on masrawy.com/Gold (4.24:1 on the live page), whose pa11y row the after run still reports on another page, and one focusable non-widget element on sabq.org's front page. The 38 unexplained rows are occurrences whose selector no longer resolves on the live page hours later (36) or whose page reloaded under the verifier (2); they are shown inside Indeterminate and are not evidence either way.
Where the removals came from
Only the classes the fixes touched. Everything else moved with the web, not with the scanner, and is not attributed.
| Rule class | Before | After | What changed |
|---|---|---|---|
| focus-indicator-invisible | 197 | 48 | focus stop centred first; a cover that is not its ancestor is 2.4.11; one row per kind |
| mobile-target-unreachable-advisory | 68 | 46 | RTL swipe reach, carousels a user can move, devices named |
| ibm-aria_hidden_nontabbable | 11 | 0 | a menu the scanner opened is closed before the page is read |
| ibm-target_spacing_sufficient | 8 | 0 | folded into target-size |
| mobile-target-size-spacing-exempt-advisory | 82 | 75 | one verdict per element across devices |
| pa11y-696811a3 | 6 | 0 | presentation strips only a table's headers and a list's items |
| target-size | 30 | 27 | 2.5.8 Equivalent exception; cross-engine twins collapse |
| heading-order | 31 | 29 | aria-level read before the tag |
| label-in-name-non-latin | 16 | 15 | Unicode names in the label-in-name probe |
| aria-hidden-focus | 10 | 9 | a menu the scanner opened is closed before the page is read |
| skip-link-missing | 37 | 36 | Arabic skip text and skip* target ids |
| ibm-aria_role_valid | 3 | 2 | every role ARIA in HTML allows on li |
| ibm-label_name_visible | 3 | 3 | an aria-hidden svg title is not visible text |
| mobile-target-size-aa-fail | 10 | 11 | same-destination link exception; empty onclick root; one verdict per element |
| document-title | 2 | 3 | access-denied and bot-check pages are not the site |
| lang-page-mismatch | 0 | 3 | new: 3.1.1 when the page's text is in another script than its lang |
| reflow-horizontal-scroll | 8 | 18 | RTL pages scroll to the left edge |
| mobile-target-size-advisory | 103 | 130 | same-destination link at 44px; skip links parked under the page |
Every site
Scores moved on 28 of 53 sites: 14 up, 14 down. A score can fall because the report got more accurate - when two engines' reports of one defect merge into a single row, the row keeps the higher severity.
| Site | Findings before | After | Score |
|---|---|---|---|
| arabbank.com | 41 | 22 | 51 → 50 |
| mawdoo3.com | 39 | 20 | 71 → 67 |
| moe.gov.sa | 43 | 24 | 49 → 55 |
| alriyadh.com | 51 | 36 | 51 → 51 |
| ju.edu.jo | 30 | 18 | 48 → 49 |
| emiratesnbd.com | 40 | 29 | 46 → 46 |
| islamweb.net | 37 | 27 | 48 → 48 |
| masrawy.com | 38 | 29 | 71 → 68 |
| aletihad.ae | 42 | 34 | 51 → 51 |
| sabq.org | 17 | 9 | 51 → 58 |
| alahli.com | 29 | 23 | 46 → 46 |
| almasryalyoum.com | 28 | 23 | 49 → 49 |
| du.ae | 45 | 40 | 47 → 45 |
| aljazeera.net | 15 | 11 | 78 → 63 |
| alrajhibank.com.sa | 28 | 24 | 43 → 48 |
| bahrain.bh | 38 | 34 | 54 → 56 |
| alkhaleej.ae | 30 | 27 | 56 → 56 |
| alqabas.com | 29 | 26 | 50 → 52 |
| dw.com | 19 | 16 | 68 → 69 |
| eand.ae | 30 | 27 | 45 → 45 |
| jordan.gov.jo | 30 | 27 | 45 → 45 |
| ksu.edu.sa | 17 | 14 | 58 → 61 |
| talabat.com | 21 | 18 | 52 → 56 |
| uaeu.ac.ae | 25 | 22 | 48 → 49 |
| jo.zain.com | 34 | 31 | 44 → 46 |
| zatca.gov.sa | 22 | 19 | 61 → 61 |
| alukah.net | 22 | 20 | 53 → 54 |
| cu.edu.eg | 24 | 22 | 50 → 50 |
| independentarabia.com | 17 | 15 | 47 → 48 |
| moh.gov.sa | 20 | 18 | 60 → 60 |
| okaz.com.sa | 31 | 29 | 51 → 51 |
| rj.com | 29 | 27 | 48 → 50 |
| royanews.tv | 30 | 28 | 47 → 47 |
| u.ae | 25 | 23 | 61 → 60 |
| alarabiya.net | 22 | 21 | 60 → 56 |
| bbc.com | 7 | 6 | 89 → 89 |
| extra.com | 24 | 23 | 51 → 51 |
| hespress.com | 22 | 21 | 56 → 56 |
| careem.com | 16 | 16 | 50 → 50 |
| echoroukonline.com | 17 | 17 | 54 → 54 |
| france24.com | 20 | 20 | 62 → 62 |
| mada.org.qa | 20 | 20 | 48 → 47 |
| amazon.sa | 30 | 31 | 47 → 46 |
| ammonnews.net | 34 | 35 | 51 → 51 |
| arabic.cnn.com | 13 | 14 | 79 → 79 |
| filgoal.com | 39 | 40 | 50 → 50 |
| skynewsarabia.com | 11 | 13 | 73 → 73 |
| ar.wikipedia.org | 32 | 34 | 49 → 47 |
| youm7.com | 23 | 26 | 52 → 50 |
| hukoomi.gov.qa | 17 | 21 | 67 → 61 |
| kooora.com | 36 | 40 | 50 → 50 |
| saudia.com | 24 | 28 | 53 → 52 |
| aawsat.com | 15 | 25 | 56 → 49 |
What this page does not claim
- Not "12% fewer false positives." Findings fell by that much. Not every removed finding was false, and the census above is the only honest breakdown we have.
- Not a false-positive rate. Producing one requires re-verifying every finding that remains, on the live page. We re-verified what changed. A single percentage would be more quotable and less true.
- Not deterministic. Live sites ship new markup, rotate carousels and block scanners between runs. 3 sites were unreachable in both runs and are excluded; a site that changed reachability between runs is excluded too.
- Not the whole story. 49 defects were found and fixed in this audit, each with a regression test proven to fail without it. Some were in the fixes themselves, found by the census. The open items are listed in the repository.
Reproduce it
- Every number on this page is computed from the audit's raw result
files by
scripts/build-proof.pyand committed with the site; none is typed by hand. - The fixture benchmark is the other half of the evidence: precision and recall on hand-annotated ground truth.
- The scanner audited here runs on every pull request through the accessibility GitHub Action
- 5,433 backend tests pass in both CI modes on the build that shipped these fixes.
- The raw figures for this edition: 2026-09-27.json. Other editions are listed in the field audit series