Game Theoretic Malware Detection

Abstract

Large software platforms (e.g., mobile app stores, social media, email service providers) must ensure that files on their platform do not contain malicious code. Platform hosts use security tools to analyze those files for potential malware. However, given the expensive runtimes of tools coupled with the large number of exchanged files, platforms are not able to run all tools on every incoming file. Moreover, malicious parties look to find gaps in the coverage of the analysis tools, and exchange files containing malware that exploits these vulnerabilities.
To address this problem, we present a novel approach that models the relationship between malicious parties and the security analyst as a leader-follower Stackelberg security game. To estimate the parameters of our model, we have combined the information from the VirusTotal dataset with the more detailed reports from the National Vulnerability Database. Compared to a set of natural baselines, we show that our model computes an optimal randomization over sets of available security analysis tools.