AMTSOlutely fabulous

The Anti-Malware Testing Standards Organization (AMTSO) was formed following a 2007 CARO workshop aimed at discussing ‘best practice’ and common flaws in anti-virus testing methods. A selection of participants from the workshop decided to join forces and in January 2008 more than 40 security software experts and anti-malware testers from around the world met to formalize the charter of the Organization. Since its inception, AMTSO has outlined its charter, held regular meetings, produced a range of standards documents and continues to work towards raising the overall standard of testing. However, there is still confusion as to what the organization does and stands for. David Harley provides his take on what AMTSO has achieved so far, and what might lie ahead.

Does Figure 1 represent your perception of the Anti-Malware Testing Standards Organization (AMTSO) [1]? Many people with an interest in anti-malware testing are now watching the organization with keen interest, but in a state of some confusion.

Figure 1. AMTSO: the view from the T-shirt.

Some see AMTSO as a group of anti-virus vendors meeting to whine about how awful testing is; others have a clearer view of who is participating, but think of it as a testing organization in its own right – or expect it to transform into a full-blown standards development organization like ISO, or a full-time compliance monitoring agency. I don’t presume to speak for AMTSO, but let me give you my own views on what has been achieved so far and what might lie ahead.

AMTSO represents a productive (and open – more members are always welcome) alliance between the anti-malware industry, mainstream testers and publishers, all of whom have had to invest much time and effort into adjusting to new threat trends. Vendors have done so by working on enhanced approaches to detection; testers, reviewers and publishers have done so by developing realistic criteria and methodologies for evaluating and comparing re-engineered technologies. The organization also benefits from the input of an advisory board [2] consisting of people who, despite their knowledge and experience of the anti-malware industry, have no vested interest in promoting it. AMTSO has always felt that the presence of this group is essential to the purpose and functioning of the organization, defending the interests of the community at large against AMTSO’s becoming a clique of self-interested vendors.

Of course, it is inevitable that vendors will be (self-)‘interested’, but most believe that they have as much to gain from a higher standard of testing across the board as anyone else [3]. In any case, an organization without the accumulated experience of the anti-malware research community would be hard-pressed to maintain credibility, as it would lack input from the people who know the most about the technology under test.

Central to AMTSO’s purpose is the recognition among the security community (including mainstream professional testers) that traditional static testing (a.k.a. throwing every available malicious program at an on-demand scanner to see how many it detects) is no longer a fully effective measure of a product’s capabilities – if it ever was.

In a threat landscape where tens of thousands of new samples [4] – most of which are non-viral (that is, trojans of some sort rather than self-replicating malware) – are seen on a daily basis, the leisurely testing methodologies of yesteryear are of limited use. However, more dynamic testing methods that reflect the complexities of a constantly shifting threatscape (and increasingly, cloud-based technologies) are themselves complex and resource-intensive. Even testers who are aware of the need to move towards dynamic testing are often deterred by the resource implications and the technical difficulties.

Clearly, there is a need for information sharing and discussion if the testing industry is to progress. AMTSO offers a forum for expert discussion as the testing industry moves towards more relevant testing methods, with input from the anti-malware sector of the security industry as well as from some of the most experienced mainstream testers.

Perhaps AMTSO’s core function is to raise the overall standard of testing, through discussion (and there’s been plenty of that!), by developing standards and documenting good practice, by encouraging the provision of tools and other resources, by providing analysis and review of tests, and, perhaps most significantly, through education. One way to raise awareness is by providing sound information from authoritative sources. To this end, AMTSO members have put together a repository of documents [5]. A particularly significant item is a testing principles document [6] that provides a high-level view of the basic rules of sound anti-malware testing. These are not tablets of stone, but documents that may be amended over time to adapt to changing circumstances and technologies.

Software testing has never been a particularly easy discipline, and security product testing poses particular challenges, since the technical aspects of attacks and countermeasures are not always well understood (or, indeed, communicated by the industry). For a long time, it seemed as though the anti-virus industry was eager to complain about bad tests, but unresponsive when asked ‘so how would you like us to do it?’ [7].

AMTSO has made moves to overturn this perception by providing copious documentation on basic principles and on specific testing issues. These don’t provide the wannabe tester with everything he could ever need in order to become a credible tester, but they do at least provide a basis for communication between AMTSO and testers (and other interested parties) currently outside the organization. It is no longer possible for anyone to claim that there is no useful, impartial information on testing to be found outside the charmed circle of vendors and mainstream testers. Even testers outside the relatively close-knit security community can draw on this resource to enhance the value of their testing, and in turn, their audiences can gain better understanding of how testing works.

Inevitably, some people have anticipated a more negative and authoritarian approach from AMTSO, and in some cases there has been disappointment that it has not been more ready to wield the stick than the carrot. There seems to be a common perception that AMTSO either has or should have set itself up as the AV industry’s police force, to monitor and enforce good testing methodologies – after all, the name of the organization includes the word ‘standards’ (not ‘guidelines’, ‘suggestions’, or even ‘good practice’). However, AMTSO is not the AV industry, and though that industry’s interests are certainly represented, they are secondary to the interests of the community at large. While there’s a place for both the carrot and the stick, AMTSO’s best course right now is to establish dialogue and consensus across the community, rather than to be the ultimate authority on good and evil in testing.

Anti-malware testing is not as easy as most people think it is (in fact all product testing is harder than most people think): it takes skill, knowledge, care and significant resources to perform a test that offers good guidance on a product’s capabilities rather than subjective opinion based on misinterpretation and unrealistic assumptions. All too often, a single data set is used by different groups to support very different conclusions.

Perhaps the best services AMTSO offers to the community in the short term can be summarized as follows:

By stressing the constructive aspects of AMTSO’s mission to improve specific methodologies (notably, hybrid and dynamic testing) and community awareness of the problems and of the solutions, the organization hopes to encourage all interested parties to follow and participate in the debate. After all, it’s to be expected that as AMTSO gains traction, it will be harder for testers and reviewers to claim credibility without demonstrating awareness of the organization’s aims and making a verifiable effort to follow them. Now is the time for other players wishing to be heard in the debate to raise their hands and voices.