David Harley and Andrew Lee ESET
The anti-malware industry has been plagued since its earliest days by one poorly designed comparative test after another. In 2007, some of the best anti-malware researchers, comparative testers, and product certification specialists took the first steps towards raising product testing standards with the formation of a group specifically focused on establishing standards and methodologies, educating both consumers and testers in discriminating between good and bad practice, and providing objective analyses of current testing practices. This paper summarizes current initiatives by the Anti-Malware Testing Standards Organization (AMTSO) and other groups, but also considers next steps, going beyond objectifying methodology, educational issues and blowing away the fog of misinformation and fallacy, to the next level.
Underlying these vital issues is a question: is it possible to make testers and certifying authorities more accountable for the quality of their testing methods and the accuracy of the conclusions they draw, based on that testing?
This paper attempts to answer that question.
