Stuck between a ROC and a hard place

Friday 6 October 11:00 - 11:30, Green room

Holly Stewart (Microsoft)

All protection products strive for perfection – we all want customers that are never infected and never experience incorrect detections (false positives). However, the era of combating malware with precise, static signatures is long gone. Anti-malware vendors must leverage next-gen methodologies, automation, and machine learning to combat the threats our customers face. Anyone who has studied machine learning knows that no model, no matter how good, will ever be a perfect reflection of the real world (it is, by definition, just a model). The trade-off between advance detection and false positives is inevitable. So, how do you make it? How do you strike the right balance for your customers?

This talk and paper will walk the audience through market data on hundreds of millions of Windows anti-malware customers and the impact of false positives and false negatives on market share. It will answer the following questions: If you have a malware miss, how likely are you to lose a customer? If you accidentally detect a clean application, are you more likely or less likely to lose a customer in comparison to a false negative? Does this vary by geography? For example, are customers in Spain more sensitive to false positives than, say, customers in China? What about customer type? Are gamers more sensitive to false positives than college students? The talk and paper will explain our methodology and insights from this in-depth, empirical research on the customer impact of false positives and false negatives.

Holly Stewart

Holly has worked in the security industry since 1997. She's held many types of roles, from technical writing in the early days, to product and program management, incident response, communications, and, for the past few years, data science. She started working for Microsoft in 2010. Currently, she works for the Windows Defender team where she manages researchers and data scientists focused on applying machine learning, automation, and other next generation capabilities to malware detection.

@ollijoi