Posted by Virus Bulletin on Apr 22, 2009
Leftfield AV detection scores used to promote ID theft tool.
With online banking an increasingly important way of interacting with our banks, the security of online transactions is a huge issue for users. Even more important for banks is the perception of security, as users worried by scare stories about stolen passwords and plundered accounts are likely to retreat to older, safer, but for the banks much more costly methods of doing business: mail, phone and face-to-face interaction.
It seems obvious, therefore, that banks should have a vested interest in reassuring their users that standard security measures - such as desktop anti-malware software - provide a decent level of protection. Some, however, have begun promoting additional tools which may help ensure secure communications - such as Trusteer's Rapport anti-phishing system, which provides extra security by countering hijacked connections and communications.
To strengthen the case for its use, Trusteer has found itself some test results which appear to show extremely low detection capabilities in leading anti-malware products - which the company claims demonstrate the need for additional protection.
In the set of test results published by Trusteer, the two biggest names in anti-malware, Symantec and McAfee, tie with a paltry score of 23%, while the highest reported score, from Kaspersky, is just 44%. The data - which totally contradicts ratings measured by numerous independent and respected testing bodies - is presented without context or further information, and some banks, heedless of the possibility of terrifying their customers, have proceeded to republish the same figures.
The lack of message clarity from the banks is compounded by the source of the detection scores. The results are produced by a group called the 'Okie Island Trading Company', generally referred to by the much more official and serious-sounding acronym OITC. Among other activities, the group publishes details of anti-malware research on its website, and does at least provide details of the methodology used in its tests, claiming that samples come from trustworthy sources and are validated.
However, the methodology has several fairly obvious flaws, not least of which is a sample selection method based on excluding samples detected by more than 25% of products - assuming that any such samples cannot represent new threats. The main objection to the methodology, however, is the use of the VirusTotal multi-scanner system to produce the results. This simplistic method of determining scanner detection capabilities not only ignores the wealth of additional protections offered by modern security software, but also presents numerous issues regarding the accuracy of the results. Since each product vendor provides VirusTotal with its own bespoke set of settings, which may or may not represent the product's best possible scanning capability, the detection results produced by the system will not be an accurate reflection of the protection provided in real-world use, even when limited to simple static scanning.
None of these potential issues are raised in the test methodology, let alone in the writeups from Rapport and at least one bank has been seen quoting the figures as fact.
The test figures suffer from several other confidence-denting problems, including conflicting dates applied by various sites quoting identical figures, and the use of click-through advertising of selected security products on the OITC site itself.
"Anti-malware testing is a complex and difficult business, which presents a lot of pitfalls for amateurs," said John Hawes, Technical Consultant at Virus Bulletin. "We work hard to ensure Virus Bulletin's tests provide valid and useful reflections of the real world, and in conjunction with groups like AMTSO we are trying to define basic standards of testing which will improve the output of all kinds of testing bodies."
Hawes continued: "It's clearly equally important to encourage people who make use of test results to select their sources carefully, and when republishing them to present data fairly and clearly. For the struggling banking sector to restore its customers' trust, and for the global economy to evolve into an online future, people's perceptions of their security when online are vital. Presenting such unreliable data, without context or explanation, can only damage those perceptions and harm the banks' business. I really don't know what their thinking is here."
More comment on the OITC tests is provided by Sophos blogger and AMTSO board member Stuart Taylor here. The OITC results and methodology can be found on its website here, with more details on the specific test quoted by Trusteer here. Trusteer's use of the figures can be seen here, with the same data presented on a troubled UK bank's website here.
Posted on 22 April 2009 by Virus Bulletin
