Richard Ford Florida Institute of Technology
Liam Mayron Florida Institute of Technology
Every year, some of the most heated discussions in the anti-malware world focus on the testing of anti-malware software. Vendors typically select tests they have done well in, and the smallest differences in detection rates are touted as absolute proof of superiority. However, while individual tests are often compared, there has been little work that attempts to analyse variance between tests and to determine from an objective and scientific perspective the importance of measured differences between product performance across tests.
In this paper, we conduct a meta-analysis of anti-malware tests for 2012 and 2013, using techniques which are commonly found in other disciplines, especially medicine. For example, statistical analysis methods, such as analysis of variance (ANOVA) can be applied to the data in order to determine if there is consistency between tests. Data mining and machine learning techniques will be applied to analyse and produce models of the data.
Using these techniques we can measure how consonant or dissonant each test is in comparison with 'average' test results. We also examine the statistical significance of differences in detection in an individual test. Is scoring slightly higher in one particular test a predictor of future performance in other tests, or are small differences simply a result of random variations? Furthermore, the manner in which test results are clustered may help identify outliers that provide different results from the majority of tests and merit further discussion. In essence, we show how meaningful rankings within anti-malware tests are, taken separately, and taken across multiple testers, as well as identify tests which seem to show differences that are not easily accounted for.
VB2013 takes place 2-4 October 2013 in Berlin, Germany.
The full programme for VB2013, including abstracts for each paper, can be viewed here.
