2007-06-01
Abstract
'Agreement was virtually unanimous that the WildList is no longer useful as a metric of the ability of a product to protect users' Randy Abrams, Eset.
Copyright © 2007 Virus Bulletin
In the fine tradition of the pioneers of the anti-virus industry, the 1st International Antivirus Testing Workshop was conceived and held in Reykjavik, Iceland last month. Michael St. Neitzel, formerly of Microsoft and Eset, now working for Frisk, had seen one too many unacceptably bad AV tests and decided it was time to bring AV researchers and testers together to try to improve the state of malware testing.
Researchers have beaten up on testers for years with little discernable result, so the notion of such a meeting to improve the status quo may seem a little quixotic unless one realizes that the opponent the AV industry faces is not a windmill, but rather, in the words of Dr Klaus Brunnstein, its Siamese twin.
The presentations were interesting and can be found at http://www.f-prot.com/workshop2007/, however the majority of value came in the discussions that followed the presentations.
A presentation modestly entitled 'Building & leveraging white database for antivirus testing' by Mario Vuksan from Bit9 was the sleeper. The presentation exposed not only the complexities of white-listing, but also that Bit9 possesses an astounding data mine concerning the rate of growth of clean software. From an industry perspective it was fascinating to find out that Bit9, one of the sponsors of Robin Bloor's paper 'AVID (Anti-Virus Is Dead)', is a power user of anti-virus software. Bloor's rant, while firmly rooted in marketing does not depict the reality of his sponsor's situation. Despite this, Bit9 may be able to contribute valuable false-positive feedback to the AV community for the benefit of users.
The hot topic of the event was the impending demise of the WildList. As Andrew Lee pointed out, anti-virus testing exists primarily for marketing. Myles Jordan of Microsoft stated that the reason the industry has hung on to the WildList for so long, and will fight to continue doing so, is because WildList testing is easy to pass. In response, VB's own John Hawes posed the question: why, if WildList testing is so easy to pass, do products in each review fail to detect all WildList samples?
Agreement was virtually unanimous that the WildList is no longer useful as a metric of the ability of a product to protect users. The WildList brought a standard of scientific repeatability and credibility to testers, however if the sentiments of test and research alike are to be acted upon, the WildList will evolve or die.
As if writing a dirge for the WildList, Verizon announced the acquisition of Cybertrust, ultimately the owner of ICSALabs and the WildList. Representatives of ICSALabs were conspicuous by their absence from the event.
While some test organizations make little or no use of the WildList, Virus Bulletin, West Coast Labs and ICSALabs are well advised to work on a plan B sooner rather than later. Speculation on what would be required for a replacement included an automated system that would not rely upon human reporters.
Testers were reminded of the paramount importance of testing malware, rather than the utter garbage prevalent in some collections. Opinions were more diverse when trying to assess what malicious samples are relevant in a test, and whether non-contextual tests against active malware are acceptable. Symantec in particular beat the drum of testing an entire suite holistically as opposed to discrete modules.
Of particular delight to many of us was the opportunity to witness the self-proclaimed swan song of respected AV testing pioneer Prof. Dr Klaus Brunnstein, who concluded the event with a history of malware testing and urged the 'Siamese twins' to go forward in a productive manner that recognizes the symbiosis between the camps. Will this workshop make a difference? Will we see improvements in testing as a result? In a highly imperfect industry that places so much emphasis on perfection, if progress has been made it is unclear whether we will recognize it - but we are, if nothing else, persistent.
Credit is due to AV-Comparatives.org, AV-Test.org, Virus Bulletin, and West Coast Labs for the courage to enter the lion’s den at dinner time!
