Changes to the VB test sets

This month's reviewer activity has been concentrated behind the scenes, with some labour-intesive changes being made in the test sets used for VB's comparative reviews.

Over the years VB's clean test sets have consisted of a reasonably representative selection of files. Recently, however, there has been some concern over the inclusion of a large number of dynamically compressed files. In most cases these are installers, which contain multiple executables under potentially proprietary encryption or compression algorithms. While a small number of these would be expected in everyday on-demand scanning, the test sets contain a far larger percentage than one would expect to encounter in a real-world situation.

With the number of scanners that contain routines for delving into such files on the increase, it has become apparent that such in-depth investigation has a severe impact on speed of scanning. Therefore, the inclusion of a large number of installer files in the clean test sets was putting the more thorough products at a disadvantage in terms of the scanning speeds we reported.

To resolve the situation, the clean executables test set has been split into two. One set contains 'pure' executables, while the other contains these dynamically compressed executables. It is hoped that this will enable a better breakdown of any future scanning speed issues.

The second major change comes in the form of a new test set – a spyware test set. Currently, there are no plans to make the detection of samples in this test set a requirement for achieving a VB 100% award.

While recognising that it is not the most stringent of test methodologies, the current plans are not to look at spyware detection capabilities on machines that are already compromised. Instead, it is envisaged that the files included in the test set will be the initial vector of the malware in question. Thus the downloaded file of a spyware application, Trojanised software with spyware functionality, backdoor servers and the like will make up the bulk of samples in the test set. As the compilation of both the spyware test set and testing methodology is still a work in progress, I would be very pleased to receive comments and suggestions at <[email protected]>.