Automated vs. Manual Accessibility Testing: What The Numbers Actually Mean

The 100% Automated Compliance Myth

Vendors selling automated widgets and scanning bots frequently promise '100% automated compliance'. In reality, automated rules engines (including axe-core, Lighthouse, and commercial crawlers) can only evaluate rules that can be determined algorithmically without human context. Extensive industry studies demonstrate that automated tools reliably identify between 30% and 40% of WCAG defects.

What Automated Scanners Do Well

Automated tools excel at catching objective, rule-based syntax errors at immense scale across thousands of pages:

• Color contrast calculations on static background/foreground text pairings.
• Missing alt attributes on img elements.
• Duplicate HTML id attributes in the DOM.
• Form controls with no associated label element or ARIA name.
• Missing lang attribute on the root html element.

The 60–70% Scanners Completely Miss

Automated scanners cannot evaluate human comprehension, dynamic context, or interactive state transitions:

• Quality of alternative text: A scanner confirms alt='dog' exists, but cannot know if the photo is actually a corporate org chart.
• Logical keyboard focus order: A scanner cannot evaluate if tabbing jumps chaotically across the screen.
• Keyboard traps: A scanner cannot detect if a user gets stuck inside an open modal dialog.
• Error message clarity: A scanner cannot determine if an error message clearly explains how to fix the mistake.
• Screen reader announcement accuracy: A scanner cannot verify whether dynamic live regions speak the right message at the right time.
• Responsive reflow usability: A scanner cannot determine if content overlaps or disappears at 400% zoom.

The Ideal Hybrid Engineering Framework

The most effective enterprise approach uses automated tests in CI/CD to prevent regressions of objective rules, while deploying experienced accessibility engineers to perform manual keyboard, screen reader, and edge-case testing before major releases.