Bin Mai Northwestern State University
Anshuman Singh, Andrew Walenstein and Arun Lakhotia University of Louisiana at Lafayette
download slides (PDF)
Malware authors are continuously probing an anti-virus system (AVS) for its vulnerabilities and developing new stealth mechanisms to take advantage of those vulnerabilities. We present a game-theoretic framework to model the strength of an AVS against such evolving offences. Game theory provides the right structure for such an analysis because it can account for both the accuracy of individual components of an AVS and also the cost of developing stealth mechanisms that take advantage of the AVS's weaknesses. The framework presented enables analytic evaluation of an AVS, and thus paves the way for the design of an optimal AVS.
The framework treats an AVS as a composition of special-case detectors (SCDs) such as MD5 checkers, X-ray scanners, heuristic behaviour-matching dynamic code emulators, etc. The composition is by means of selector logic that determines which SCDs are invoked on a given sample. By attaching costs and pay-offs for the attacker and defender, game-theoretic analysis can be performed. Using this framework we show that the compositions are beneficial only when the cost of developing stealth techniques is above certain model thresholds. We also show that, surprisingly, when stealth design is easy and selector accuracy is high, the difference in detection rates of the SCDs should be low for optimal performance of the AVS.
