A Change of Heart by Sitemorse?
Sitemorse has again come under fire regarding its automated testing tool and the publication of ‘ league tables’ based on results obtained with this tool. The main focus of the complaints was that attempting to rank accessibility using an automated tool alone was, at best, highly confusing and, at worst, potentially harmful. Only about 40% of the WCAG 1.0 Guidelines can be tested using automated tools such as Sitemorse’s. Therefore attributing a percentage accessibility score to a site using a tool that can only check for less than half of the relevant checkpoints is a recipe for confusion and misunderstanding.
Also, the WAI Guidelines include a number of checkpoints which contain conditional clauses such as Until user agents allow…
and Until user agents can…
and, as such, are open to interpretation. Applying absolute pass or fail scores based on automated test results – as Sitemorse does – is therefore, by definition, a non sequitur.
Having resolutely ignored these issues thus far, it would appear that Sitemorse has had a possible change of heart according to a post on Public Sector forum on July 19.
Nevertheless it was decided greater sense could be made of the accessibility rankings in future if the SiteMorse percentage scores reflected failing checkpoints rather than passes (because, as we’ve said, even if a testing tool does pass a particular checkpoint it doesn’t in itself necessarily indicate compliance).
With this in mind SiteMorse has agreed from August to report the accessibility element of its league tables based on percentage failures as opposed to percentage passes, incorporating this into the overall aggregated scores.
However, this may not be as greater change as it appears to be.
It is hard to see how reporting percentage failures will be any more helpful than reporting percentage passes. Reported percentage anything based on unverified results from a tool that can only test against 40% of the WAI Guidelines is still likely to create misunderstanding and may contribute towards a false sense of security in some cases.
Either way, the next published league table from Sitemorse should make for interesting reading.