Automated testing - How useful is it?

Contributed by Grant Broome

The truth about automated testing

There are a few good tools on the market for assessing potential technical flaws in the accessibility of a website. The author shares his experience of working with a number of these tools and asks the question "How useful is automated testing?"

The benefit of automated tools

First off, let's make one thing very clear. Automated testing tools can work. Used properly, they will help to cut an auditor's workload in half. That's because automated testing tools identify around half of the problems with web pages. The other half is down to the auditor who will need to be skilled enough to identify the problems that the auditing tool misses and verify that the auditing tool is assessing the site correctly. Many who purchase automated tools believe that they make entirely accurate assessments of web pages. In actual fact, automated testing tools are prone to making mistakes, some of which are discussed in this article.

I strike quite a critical tone through this write up, simply because it seems that lots of people see automated tools as a do-it-all solution and some vendors while stating that automated tools aren't a complete solution, actually sell them as if they are. One such vendor actually publishes a league table based purely on automated testing results leading customers to believe that the status of a website can be determined by the automated test alone! This is far from being true and I hope to be able to address this by clearly illustrating that any automated testing tool can be useful, but it is only a part of the solution.

The problem with automated tools

Incomplete results

Automated tools just aren't smart enough. The general consensus from those who have used automated testing and are familiar with its capabilities and limitations agree that automated testing can never be a complete solution for informing a web manager about the status of his site. During the course of researching for this article it was discovered that not a single Priority 1 checkpoint (Priority 1 being the most important level of accessibility) could be fully checked by automated testing tools. Here's an analysis of each priority 1 checkpoint that illustrates how each priority 1 checkpoint to a greater or lesser degree needs to be checked manually.

In General (Priority 1)

Checkpoint 1.1 Provide a text equivalent for every non-text element

While it is possible for a machine to check whether an image has any sort of text alternative, it is almost impossible for the machine to be able to determine whether that alternative is adequate. A web author could give an image of a dog the text alternative of "cat" or even "archive image" and the machine would not be able to detect the error.

Checkpoint 2.1 Ensure that all information conveyed with color is also available without color, for example from context or markup.

Let's take an example of an online store with a list of products on a page. The list items in green indicate that the product is in stock. The list items in red show that the product is out of stock. There is text on the page that explains what the colour indicates.

• 001 Leather belt (large) £15.95
• 002 Leather belt (medium) £14.95
• 003 Leather belt (small) £13.95

The human tester would easily spot that the list is flawed because it is only usable by those that can distinguish between red and green, excluding those who are colour-blind or who are blind and depend on a special technology to read them the page. The machine would normally miss this major fault and indicate that the page was valid.

Checkpoint 4.1 Clearly identify changes in the natural language of a document's text and any text equivalents (e.g., captions).

While it is theoretically possible for a machine to be able to identify language changes on a written page, the reality is that none of the major automated software checking tools have any idea that "such is life" is English, and "c'est la vie" is French. In fact, A tool would need to support a multilingual dictionary in order to offer this capability, and none of them do.

Checkpoint 6.1 Organize documents so they may be read without style sheets. For example, when an HTML document is rendered without associated style sheets, it must still be possible to read the document.

There is no automated testing tool on the market that can tell whether the reading order of the page is affected by style sheets. You could create a page that was perfectly readable while style sheets were supported and yet make no sense at all when they are not. The machine, once again would not be able to inform you of this catastrophic failure.

Checkpoint 6.2 Ensure that equivalents for dynamic content are updated when the dynamic content changes.

There is no tool available that is able to check this. It is a complicated checkpoint that requires human testing.

Checkpoint 7.1 Until user agents allow users to control flickering, avoid causing the screen to flicker.

While some tools do make a best guess at flickering images, the results are never accurate and require manual checks.

Checkpoint 14.1 Use the clearest and simplest language appropriate for a site's content.

There are tools and algorithms that can check the readability of a piece of text, but again the result is simply a guide. It is unable to detect who the intended audience is or whether the language is appropriate for that audience.

And if you use images and image maps (Priority 1)

Checkpoint 1.2 Provide redundant text links for each active region of a server-side image map.

For a tool to be able to verify text link alternatives for active regions of a server side image map are valid, it would first need to know what the server-side links were and whether the links carried accurate descriptions. This is a task that can only be done manually.

Checkpoint 9.1 Provide client-side image maps instead of server-side image maps except where the regions cannot be defined with an available geometric shape.

This guideline is intended to advise the developer to use client-side maps wherever possible. It cannot be checked automatically.

And if you use tables (Priority 1)

Checkpoint 5.1 For data tables, identify row and column headers.

A tool can only check if the row and column headers are present. It cannot check if they are appropriate and in this regard, the principle is similar to checkpoint 1.1.

Checkpoint 5.2 For data tables that have two or more logical levels of row or column headers, use markup to associate data cells and header cells.

The success of this checkpoint relies on accurately assessing checkpoint 5.1 above, which cannot be assessed automatically.

And if you use frames (Priority 1)

Checkpoint 12.1 Title each frame to facilitate frame identification and navigation.

The tool can only check if the frame identification is present. It cannot check if it is appropriate.

And if you use applets and scripts (Priority 1)

Checkpoint 6.3 Ensure that pages are usable when scripts, applets, or other programmatic objects are turned off or not supported. If this is not possible, provide equivalent information on an alternative accessible page.

Tools do not measure whether the site is usable with scripts disabled. Many tools will flag up an error if a script is found on a page even though the script may be a simple button rollover (something this simple does not need an alternative, but an error is recorded by the automated tool regardless).

And if you use multimedia (Priority 1)

Checkpoint 1.3 Until user agents can automatically read aloud the text equivalent of a visual track, provide an auditory description of the important information of the visual track of a multimedia presentation.

No tool is capable of viewing a movie or animation and reporting on whether an appropriate auditory description has been provided. This is a complex process that can only be assessed manually.

Checkpoint 1.4 For any time-based multimedia presentation (e.g., a movie or animation), synchronize equivalent alternatives (e.g., captions or auditory descriptions of the visual track) with the presentation.

Very similar to checkpoint 1.3, automated tools are far from being able to detect whether the movie or animation is accessible. This can only be done by an experienced accessibility specialist, or testers with specific disabilities.

And if all else fails (Priority 1)

Checkpoint 11.4 If, after best efforts, you cannot create an accessible page, provide a link to an alternative page that uses W3C technologies, is accessible, has equivalent information (or functionality), and is updated as often as the inaccessible (original) page.

It stands to reason that this checkpoint could not be satisfied unless each and every checkpoint preceding it can be accurately measured.

In conclusion of this examination it was found that none of the 16 priority 1 checkpoints can be accurately tested using automated tools alone. This is a fact that can be demonstrated using any tool currently on the market.

Given that many of these automated reports are handed over without the client being made aware that they are demonstrably incomplete, it is easy to see how a developer could read an automated report and feel satisfied that his site is fully accessible when in fact it is not. In some instances, a developer may be told that his site was inaccessible, when in fact it is already fully compliant with WCAG guidelines.

False positives

In addition to errors that can't be checked, automated tools frequently falsely detect errors. A link to www.qantas.com.au will often be detected as an audio file because it has a ".au" extension. This will cause the page to fail because there is no text alternative for the audio file.

On occasion a tool will report that an alternative text is dubious. A tool may detect that "123.gif" is not a valid alternative text for an image. In most cases you would check the image and find this to be a correct assessment, but occasionally the error is falsely reported. In the preceding example, the alt text could be for an image of numbers that is intended to be used by teachers for learning purposes as part of a multimedia presentation. The ".gif" extension included in the alt text may be important information that needs to be included in the alt text.

A recent example

Recently the GAWDS site was reported to have failed an important priority 1 checkpoint (1.1). It was claimed that an image on the GAWDS website carried an alt attribute (used to identify images to screen readers among other things) that was incorrect. The implication was that this made the entire site inaccessible.

The image in question had the alt text "GAWDS logo - master.png". The automated tool identified the .png in the alt text and an algorithm detected that the author had simply used the filename for the image alternative. This particular instance of the image can be found on a page with many other images. The .png information is included in the image alternative so that a visitor can distinguish it from another similar image, such as a .jpg.

The computer took a guess, and it was wrong. The test claimed that the GAWDS site was inaccessible but it had made a simple mistake. This is a good example of what happens when automated site reports make absolute statements that need human verification.

Fooling the tool

It's simple to trick an automated tool into thinking that your site is accessible, and many web developers have fallen into this trap. Some web developers don't even realise that they do it. Their knowledge of accessibility is limited to what the tool teaches them. Another reason is pressure to conform.

A company in the UK publishes league tables which ranks websites for accessibility based purely on the results from an automated test. A little while ago I spoke to a web developer of a public sector organisation who was under pressure to lift his web page in the rankings of this league table. To accomplish this, the developer took out all the heading elements from his entire website. He did this because the results he had been given told him that his headings were in the wrong order. The tool does not check to ensure that headings are being used appropriately, neither does it detect that headings are missing completely. The result of removing the headings had the effect of making his website shoot up the league table, but resulting in his website being far less accessible than it was before.

It's not wise to base your accessibility performance on league tables. The results are based on an overly simplified automated test which is ultimately flawed and easy to fool.

How to Use Automated Testing Tools

The first step towards using automated testing tools efficiently is to understand that they can't do everything. They can help you identify areas where your site may not be up to scratch, and they are very good at spotting missing elements in pages such as labels for forms. Without labels, forms can be very difficult for assistive technologies to identify properly; automated tools are good at recognising these omissions in this and similar scenarios.

What automated tools can't do is measure subjective checkpoints such as the appropriateness of alt text or even the relevance of a form label. Because of this, they cannot tell you if your website is accessible. They can only be used to inform you if your web site has passed certain elements of certain checkpoints.

As demonstrated earlier in this article they are potentially harmful if they are presented to a client or web author with no additional support or a verified manual audit. They should never be presented as factual documents because the evidence shows that they frequently contain errors.

It should be fairly clear that automated tools cannot fully assess your website, but all is not lost. Site auditing is still an important process which benefits many users.

Automated tools still have a valuable part to play when used properly but rather than relying on the results of automated testing alone, you have a number of good alternatives:

Learn how to identify accessibility issues manually

Learning how to manually check a website for accessibility is a rewarding and worthwhile skill. You will need to acquire knowledge of the Web Content Accessibility Guidelines (WCAG), and gain an understanding of how disabled people use the web. There are many forums that can help you do this, notably accessifyforum where lots of accessibility advocates will be happy to offer advice and guidance.

In addition, you'll need to get to know your tool of choice. Each Automated testing tool will have its strengths but also some quirks. You'll need to learn when you results need to be checked manually to ensure you're getting the right information.

Take advice from a third party who specialises in web auditing.

There are a number of high profile organisations that work in this area. If you decide to take advice from a third party then make sure that you ask for references from previous clients and examples of their previous auditing work. Web Accessibility is a buzzword at the moment and everyone seems to be an expert. Make sure you're getting the right advice.

Have your site audited by disabled people.

This is the true test of accessibility. A pan-disability service like that offered by the Shaw Trust will help you to ensure that your website is accessible to the widest possible audience, and it's the only way of being sure that your site can be accessed by users with a range of disabilities.

Regular audits by a third party with a track record of providing high quality auditing services may in the long term save both time and money. Consider having your site audited by a reputable company and make absolutely sure that automated results are supported by a manual report and pan-disability user testing.

Choosing an automated tool

There are a good number of tools on the market that can be installed on your desktop or server and be used to perform tests on a regular basis or when required. Tools tend to be fairly similar in the way they check for errors so just make sure that the results are presented in a way that is informative and easy to understand.

Avoid services where you are just sent a monthly automated report. Some services on offer simply offer you a monthly reading of your website performance. One obvious drawback is that this may not be regular enough for your purposes or you may need to do a spot check following some recent work. The other drawback is that you'll never really get to learn the quirks of the tool that is sending you the results.

Make sure that you pick a tool that allows you to run your own tests as and when you need to.

Conclusion

Web auditing tools can be useful when used properly. They can be used to drastically reduce workloads for web developers but they won't always get it right and a firm knowledge of web accessibility issues is essential.

Regular random checks on test results and a good understanding of the capabilities of your chosen tool will help to ensure that your test results are accurate, and never rely on automated testing results alone.

List of Automated testing tool vendors

• SSB Technologies InFocus
• HiSoftware AccVerify
• Watchfire
• Accessibility Valet

If you are on a tight budget and have just a few pages to test, then there are a number of tools that can be used online for free:

• GAWDS list of web accessibility evaluation tools
• NILS Accessibility toolbar for Internet Explorer
• Chris Pedrick's WebDeveloper
• A list of additional resources compiled by WATS.CA(Added 11th July 2005)

Comments

Comments are not available at the moment due to spam problems.