Rebuilding testing for the future

Igor Muttik and James Vignoles McAfee

This presentation discusses several aspects related to testing the ability of security products to detect malware. The complexity of malware and of the security solutions go up really quickly and we present arguments as to why we believe that comprehensive QA is no longer viable and why a switch to a more statistical approach is in order.

We look into the problem of compiling a representative 'next-generation' sample test set:

balancing the test speed with the breadth and depth of testing
ranking threats and removal or downgrading the rank of legacy threats (e.g. DOS and Word6 viruses)
removal of short-lived and inactive threats (e.g. spammed downloaders where the site was shut down)
tracking the history and relationship of malware samples (downloader of what? where from? is the URL still alive? gaming password stealer for WoW or for Zhengtu?)
excluding most HTMLs (are encrypted URLs malicious code or are they just obfuscated data?)
downgrade or exclude downloaders for the sake of what they download?
ranking clean data and false alarms (just like malware clean programs are very much not equal)
better separation of the malware samples and spam (encrypted URLs could be tricky to classify)
fair representation of local threats (e.g. could there be too many Brazilian password stealers vs oriental trojans related to gaming?)

We present a topological and percolation model of malware distribution and present arguments as to why the user profile should be part of the test.

We discuss potential solutions to QA problems:

running different tests for different user profiles
organizing collections in 'attack sample groups' rather than individual samples
collecting telemetry data via 'testing/reporting' plug-ins to security products
using live telemetry to collect malware execution data (frequencies, geo-location information, etc.)
using telemetry to rank malware attacks
standardizing the format of telemetry data and sharing it within the industry
testing complete security products (e.g. AV bundled with anti-spam, rather than pure AV)