Why there’s no one test to rule them all

Anti-malware products are all alike the world over – with the same tactics, usage, features, speed of updates and target market, right? If that were true it would stand to reason that there would be only one or two types of appropriate tests to put those products through their paces. Just running a large number of threats and clean items against the different companies’ products would be sufficient. In reality, though, that is not the case.

It’s my position that there is no ‘One Test to Rule Them All’. The overarching objective of all tests is to emulate what users do in the real world. But users in China will have a different set-up from those in Germany, just as users in major banks will differ from home users with mobile anti-malware products. The threats that affect them differ, as does the information they want.

Similarly, the consumers of tests have interests in different types of products as well as different information. Anti-malware vendors themselves are consumers of tests. Their interests are similar in many ways to those of a user, but not identical. (After all, there is no financial incentive for users, regardless of a test’s outcome.)

So what should testers be doing? First, I believe there is still value in what are now considered ‘traditional’ testing methods. Especially with new and emerging markets (both geographically and technologically), periodic static testing can function as a baseline to indicate which solutions are valid anti-malware products. There may come a time when anti-malware scanner technology has changed so much that this is no longer adequate, but until then static tests remain a good way to validate basic functionality.

Beyond that, things get more complex. While there is a lot of the traditional technology in modern anti-malware products, there are also a lot of new modules and features. While most folks agree to a certain extent on what an anti-malware product looks like, not everyone agrees what constitutes newer technologies. Testers must often make decisions regarding what qualifies as a Standard Newfangled Widget when different vendors come up with different ways of going about things. Anti-spyware and anti-spam are excellent examples of how this has played out in the past. Testers had to make decisions, with a significant amount of input from vendors, as to what samples were appropriate and how they needed to be addressed. Technologies like IPS/IDS or DLP make this more complicated still, as they bear less resemblance to signature scanners.

Because of the speed and prevalence of malware, time is one of the most essential elements. Scans on users’ machines don’t happen only quarterly or monthly, so the frequency of tests has increased. As the testing time decreases, the relevance of samples becomes vastly more important.

People don’t only use on-access or on-demand scanners, but also run-time detection such as behavioural scanners and emulators. Most people in the anti-malware industry these days agree that dynamic testing is essential.

Different testers may also choose to validate detection in various other ways as well. For example, retrospective testing examines scanners’ abilities beyond simply detecting malware which is already known. Those products with exceptional heuristic or ‘generic’ detection capabilities can differentiate themselves here.

There are also concerns which go beyond the accuracy of detection, but which are nevertheless important to users. Performance testing in the sense of memory/CPU usage can reassure users that, during scanning, their machine will not be disproportionately affected – they can see that they don’t need to sacrifice usability for thoroughness of protection.

Because every product has strengths and weaknesses, having a variety of different tests is essential. You must have a wide and varied vocabulary to describe things to people in a way that is meaningful to the majority. Let us not limit our vocabularies to just a few adjectives, but strive to serve and create an erudite user base.

Latest articles:

Bulletin Archive