Accuracy you can audit.
We publish our benchmark so you never have to take our accuracy claims on faith. Hand-annotated ground-truth fixtures - including false-positive traps built from real-world noise (screen-reader-only markup, hidden skeletons, third-party widgets) - with precision and recall computed per fixture and averaged macro-style.
Precision
100%
Of what we flagged, how much was real
Recall
88%
Of what's real, how much we caught
F1
92%
Harmonic mean. The honest single number.
Scanner: Accessibility Pro v2.15.0 · 15 fixtures · Ran 2026-08-27
Measured on the full pipeline (filters, confidence scoring, dedup, caps) - per-row finding precision 97%, 1.41 rows per real defect (1.00 = no duplicate inflation).
Per-fixture breakdown
Each row shows which WCAG criteria were expected (ground truth) vs which the scanner detected. False positives and false negatives are listed explicitly. We do not average them away.
| Fixture | Expected WCAG | Detected WCAG | Precision | Recall | F1 |
|---|---|---|---|---|---|
01-missing-alt Missing alt on content image | 1.1.1 | 1.1.1 | 100% | 100% | 100% |
02-form-without-label Form input missing label | 1.3.1, 3.3.2, 4.1.2 | 1.3.1, 4.1.2 | 100% | 67% | 80% |
| False negatives: 3.3.2 · scanner missed ground-truth criteria. | |||||
03-contrast-fail Low contrast body text | 1.4.3 | 1.4.3 | 100% | 100% | 100% |
04-duplicate-id Duplicate id attribute | 4.1.1 | 4.1.1 | 100% | 100% | 100% |
05-missing-title Document missing <title> | 2.4.2 | 2.4.2 | 100% | 100% | 100% |
06-empty-button Button with no accessible name | 4.1.2 | 4.1.2 | 100% | 100% | 100% |
07-heading-skip Heading hierarchy skips a level | 1.3.1, 2.4.6 | 1.3.1 | 100% | 50% | 67% |
| False negatives: 2.4.6 · scanner missed ground-truth criteria. | |||||
08-empty-link Link with no accessible name | 2.4.4, 4.1.2 | 4.1.2 | 100% | 50% | 67% |
| False negatives: 2.4.4 · scanner missed ground-truth criteria. | |||||
09-no-lang Document missing lang attribute | 3.1.1 | 3.1.1 | 100% | 100% | 100% |
10-aria-invalid-idref aria-labelledby points at missing id | 1.3.1, 4.1.2 | 4.1.2 | 100% | 50% | 67% |
| False negatives: 1.3.1 · scanner missed ground-truth criteria. | |||||
11-clean Well-formed page with no violations | … | … | 100% | 100% | 100% |
12-combined Multiple violations on one page | 1.1.1, 1.3.1, 1.4.3, 4.1.2 | 1.1.1, 1.3.1, 1.4.3, 4.1.2 | 100% | 100% | 100% |
13-sr-only Screen-reader-only content done correctly (FP trap) | … | … | 100% | 100% | 100% |
14-hidden-skeleton Hidden skeleton / template markup (FP trap) | … | … | 100% | 100% | 100% |
15-third-party-widget reCAPTCHA + portal markup, framework-managed (FP trap) | … | … | 100% | 100% | 100% |
Methodology
Every fixture is a self-contained HTML file with hand-annotated
ground truth. Comparisons are made at the WCAG criterion level rather than by engine-specific rule ID. This lets
axe-core's image-alt and IBM Equal Access's WCAG20_Img_HasAlt both count as valid detections
of WCAG 1.1.1.
The overall precision / recall / F1 numbers are macro-averaged: each fixture contributes equally, regardless of how many violations it contains. This penalises scanners that do well only on high-volume fixtures. Micro-averaging (weighted by violation count) would inflate the numbers; we use the less flattering option on purpose.
Clean pages (zero expected violations) count as precision 100% + recall 100% if the scanner emits nothing. Any false positive on a clean page drives precision to zero. This is the strictest test and catches trigger-happy scanners.
Fixtures are controlled; the web is not. The field audit of real websites runs the same scanner through production against live sites, before and after a round of fixes, and re-checks every removed finding on the live page.
Reproduce or challenge our numbers
- The fixtures, the hand-annotated ground truth and the runner
(
python benchmark/runner.py) ship with the codebase, so the whole table can be regenerated from source rather than trusted. - If you think an expected criterion is wrong, say so: the ground truth is a JSON file, and a disputed row is a one-line change.
- The engines measured here run on every pull request through the accessibility GitHub Action
- Numbers on this page always reflect the latest committed run. The fixtures and expected criteria are listed above so a disputed row can be checked by hand; the harness itself is not public yet.