The (testing) world turned upside down

Thursday 5 October 11:00 - 12:30, Small talks

David Harley (ESET)
John Hawes (AMTSO)

We often hear that anti-virus is dead, but if that is really so, where does it leave anti-malware product testing?

After decades of slow progress, security product testing has been moving away from the chaotic practices of the early 90s, to models of better practice as to some extent codified in the AMTSO 'Fundamental Principles of Testing'. Yet we've recently seen a resurgence in approaches to comparative testing that have long been flagged with a red light:

Disabling of layers of functionality and the demotion of whole product testing
Simulation as a comparative testing tool
Malware creation
Opaque sourcing, selection, classification and validation of samples
Promotion of D-I-Y testing as superior to independent testing.

Have so many of the assumptions made on both sides of the vendor/tester divide been wrong all along? Or is just this another instance of The (Testing) World Turned Upside Down by marketing?

In this paper, we re-examine those assumptions, set in the context of:

The good, the bad and the ugly in early product testing, and the slow-burn reaction of the security industry, culminating in the CARO testing workshop and the first steps towards the foundation of AMTSO.
The painful evolution of AMTSO into a source of testing guidelines and somewhat less reliable mediation between the opposed yet interdependent testing and vendor communities.
VirusTotal's re-engineering of its policies, and the impact on AMTSO of the subsequent semi-assimilation of self-named 'next-gen' vendors into its membership.
A new generation of conflicts between vendors and testers.
The claimed divergence in anti-malware technologies and mindset across the spectrum of mainstream and newer vendors. Does this divergence necessitate new testing methodologies? How can such methodologies be appropriately evolved, and how successfully can AMTSO play its part?

Or are both AMTSO and mainstream independent testing doomed to failure and fragmentation?

The presentation of this paper by ESET Senior Reseearch Fellow David Harley will be followed by a discussion among the audience led by AMTSO's John Hawes.

David Harley

David Harley is a security researcher, author and editor. His academic background is in social sciences and computer science. From 1989 to 2006 he worked in medical informatics, specializing in security and data protection. Since setting up the Small Blue-Green World consultancy in 2006, he has worked closely with ESET, where he is a Senior Research Fellow. He has authored, co-authored and/or edited around a dozen security books, including Viruses Revealed and the AVIEN Malware Defense Guide. VB2017 sees his 16th Virus Bulletin paper, which is probably enough for one lifetime. He claims to be semi-retired but remains obsessed with the psychosocial elements of security and the lack of it. His leisure time is mostly devoted to composing music and playing guitar.

John Hawes

John Hawes has been involved with anti-malware testing since 2000, first spending over five years in the QA lab at Sophos before joining Virus Bulletin in 2006. For over ten years he ran VB's testing operations, including the renowned VB100 award scheme, producing over 60 comparative reports, and for three years took joint responsibility for running the company. During that time he joined the Board of Directors of the Anti-Malware Testing Standards Organization (AMTSO), serving as Chair in 2015-16, and in 2017 he left VB to become AMTSO's Chief Operating Officer, as well as a consultant and writer at Tick Tock Social Ltd.