Imagine if, every Thursday, your shoes exploded if you tied them the usual way. This happens to us all the time with computers, and nobody thinks of complaining.
Automated Accessibility Testing
There has been a great deal of discussion recently on the role of automated accessibility testing software. Much of the discussion has centred upon whether these tools are really useful and whether they do more damage than good.
First of all, it should be pointed out that all automated testing software has limitations. Every single tool has its bugs and blindspots. There isn’t a tool on the market that doesn’t have the potential to report false positives (i.e. highlight errors where none exist) , and to report false negatives (i.e. completely miss genuine accessibility barriers).
The main reason for this is pretty obvious. Software is stupid. It is as thick as two short planks. If you encourage it to be too sensitive, it will flag up so many false positives that your testing workload might actually be increased (remember that the reason for many of these automated tools is to reduce workload). Reduce the tool’s sensitivity by too much and you risk the chance of missing serious accessibility errors – unless of course you intend to manually check every page which pretty much negates the reason for running such a tool in the first place.
So it stands to reason that a useful tool needs to strike a logical balance between reporting too many false positives and missing too many genuine issues. Most of the tools on the market appear to do just that. However, every one of these tools should come with the qualifier to the effect that their results need to be manually verified before they can be reported with any degree of confidence.
Again, most web developers seem to understand this. Very few of them would accept the results from an automated testing tool without argument. They understand that an automated testing tool is just that – a tool. No more and no less. It cannot replace manual verification or user testing and it should never be used in isolation. Grant Broome of the Guild of Accessible Web Designers has authored an excellent document on the usefulness of automated testing in relation to testing for Priority 1 issues, so I won’t repeat any of his arguments here.
The problems, however, really start to arise when non-technical managers are presented with reports based on purely automated testing. They are not in a position to manually verify these results and may not fully understand the limitations of such testing.
It would seem to me, therefore, that the responsibility for ensuring that automated testing results are not taken at face value rests upon the publishers of such results. They should stress that any results from purely automated testing must not be taken at face value. A vague qualifying addendum about the merits of manual verification and user testing simply isn’t enough. Unfortunately, the commercial providers of some automated testing tools seem to think that this would damage their commercial viability and, consequently, appear to downplay the dangers of trusting automated testing results and any associated ‘ league tables’.
It is not so much that their software is at fault as the manner of their reporting – which at times, borders on the highly irresponsible.
Caveat emptor! (buyer beware!)