Your Scanner Says 98%. Why Did the Audit Still Fail?

A vendor ran a scan across your site. The dashboard came back green — a handful of colour-contrast warnings, a few missing alt attributes, nothing alarming. Six weeks later a real evaluation, or a complaint, surfaces problems the scanner never mentioned: a form nobody can submit with a keyboard, a table that reads as gibberish, alt text that describes the wrong thing entirely.

The scanner was not broken and the vendor was not lying. Automated tools are extremely good at a specific class of problem and structurally incapable of finding the rest. Knowing exactly where that line falls is what stops you from buying false confidence.

What can a machine actually check?

Automated testing is reliable wherever a requirement can be settled by inspecting code without understanding meaning. That includes a great deal of genuinely useful ground:

• Images with no alt attribute at all
• Form inputs with no programmatic label
• Text colour-contrast ratios against a solid background
• Missing page language declarations
• Empty links and buttons, duplicate id values, invalid ARIA roles and attributes
• Heading levels that skip, tables with no header cells, missing document titles
• Whether a PDF has a tag tree, a declared language, and a marked title

That is not a trivial list. Scanners run in seconds across thousands of pages, they never get bored on page 400, and they catch regressions the moment a developer introduces one. Anyone who tells you automation is worthless is overcorrecting.

The honest measure of the gap comes from the tool vendors themselves. Deque, which builds the widely used axe engine, has published that its automated testing found issues for 16 of the 50 Level AA success criteria in WCAG 2.1 — roughly 30% of the criteria — while arguing that because some issue types are far more common than others, automation catches a larger share, around 57%, of the total volume of issues found. Both numbers are from the same research, and the tension between them is the point: even on the most favourable framing its own maker publishes, close to half the problems are not detected.

What a machine cannot decide

The undetectable half is not a random remainder. It clusters around one thing — meaning.

Is the alt text correct? A scanner confirms the attribute exists. It cannot know that your organizational chart is described as “chart” or that a photo of a flooded street is labelled “image1.jpg”. Alt text that is present and useless passes every automated check ever written.

Is the reading order sensible? A tool sees a sequence. It cannot tell that the sequence puts a table’s footnote before its heading, or that a two-column newsletter interleaves the columns line by line.

Does the keyboard path work? Automation can flag an element that is unreachable. It cannot tell you that the focus ring vanishes against your navy header, that a modal traps focus and never gives it back, or that a dropdown opens on hover only. These are found by a person putting the mouse down and using Tab.

Do the headings describe the structure? A scanner verifies levels do not skip. It cannot judge that all fourteen of your headings say “Information”.

Are captions accurate? Automation detects that a caption track is present. Accuracy, speaker identification, and whether the captions match the audio require someone to watch.

Does an error message help? A tool sees that an error is announced. Whether “Invalid input” tells a person what to do about it is a judgment call.

Is the whole task completable? The most important question of all. Automated tools test pages; people complete processes. A permit application can pass on every individual screen and still be impossible to finish end to end.

There is a second, quieter problem: false positives. Scanners flag things that are fine — decorative images correctly marked as artifacts, contrast measured against a background the tool guessed wrong. Teams that spend a quarter chasing phantom issues learn to distrust the tool, and then start ignoring the real findings too.

How does this apply to documents?

The same split exists for PDFs, and it is sharper there.

The Matterhorn Protocol, the PDF Association’s testing model for the PDF/UA standard, breaks the requirements into 31 checkpoints and 136 failure conditions — and states plainly that 89 of those conditions can be checked by machine while 45 require human validation. That is a formal, published acknowledgment, written into the testing methodology itself, that roughly a third of PDF accessibility cannot be automated. Anyone selling fully automated PDF compliance is selling past a number their own standard body printed.

Our PDF accessibility checker is deliberately honest about which side of the line it sits on. It reports the structural facts about a file — tags, language, title, reading order signals — quickly and for free. It does not, and cannot, tell you whether the alt text is right.

The multilingual case automation cannot touch at all

If you publish in more than one language, the gap widens considerably.

A scanner can verify that a language is declared. It cannot verify that the declared language is the language actually on the page — a Spanish document tagged as English passes the check and fails the user. It cannot tell that a passage of Haitian Creole inside an English notice is untagged, which is WCAG 3.1.2. It cannot tell that your Arabic PDF’s reading order is reversed, because the tag tree is internally consistent and simply wrong. And it cannot evaluate translated alt text, because that requires reading two languages and comparing them.

Every one of those needs a person who speaks the language and knows what a screen reader is supposed to do with it. That combination is rare, and it is the specific reason we do this work. The failure modes are set out in translated documents and WCAG language of parts.

So what should you actually buy?

Use both, for different jobs.

Automation is your continuous layer. Run it in the build pipeline so regressions are caught the day they are introduced. Run it across the whole site to find the easy, high-volume problems cheaply. Use it to prove you have not gone backwards since the last audit.

Manual evaluation is your assurance layer. Scope it to a representative sample — your templates, your highest-traffic pages, and every page in a transactional path from start to finish. Include keyboard-only testing, screen-reader testing, and a human reading of alt text, headings and error messages. Get findings written against specific WCAG success criteria by number, so remediation is unambiguous and so the results can support a conformance report later.

The order matters. Automate first to clear the volume, then send a human at what remains. Sending an expert to hunt for missing alt attributes wastes the expensive resource on the cheap problem.

And be careful what you accept as evidence. A scan report is not an audit, and neither is a conformance claim built only on scan output — which is exactly the trap we describe in what a VPAT actually is. If a proposal offers “automated testing against WCAG 2.1 AA” with no manual component, it is quoting a third of the job.

Want to know which layer you are missing? Tell us your site’s page count, the number of documents you publish, the languages you publish in, and whether anything has ever been tested by a person — then request a quote. We will tell you what to automate, what genuinely needs a human, and what a defensible evaluation of your site would cover. Our accessibility assessment page explains how we structure the manual pass.