Say hi to malware - using a deep learning method to understand malicious traffic

Friday 6 October 11:30 - 12:00, Red room

Zhaoyan Xu (Palo Alto Networks)
Tongbo Luo (Palo Alto Networks)
Wei Xu (Palo Alto Networks)
Kyle Sanders (Palo Alto Networks)
Xin Ouyang (Palo Alto Networks)

Recently, thanks to the exponential growth of data size in our daily communications, it has become more and more challenging for security practitioners to identify little drops of malicious traffic among the sea of benign data. In particular, current advanced persistent threat (APT) attacks commonly spread their communication across multiple independent network sessions, making it hard for the traditional IPS signature generation scheme to succeed. Similar to learning a foreign language, without understanding the syntax, semantics and context of malicious communications, it is almost impossible to defend against them.

In this paper, we attempt to take a deep look into the cross-session communication of malware, and to understand their language automatically. To achieve this, we employ multiple deep learning methods to systematically analyse the syntax, semantics and contextual information of the malware's communication. In detail, we split each malware family's network communication into words, packets, and sessions. Then we develop a multi-layer recurrent neural network to describe the internal logics of each malware dialect. Based on the learned model, we can generate highly effective intrusion prevention signatures without any manual effort. Furthermore, we have developed a deep enforcement learning method to handle the variants of malicious traffic. Our method can automatically generate signatures for over 40 malware families. By evaluating millions of live traffic data, our signatures can detect malicious traffic without any false alarms.

	Zhaoyan Xu Zhaoyan Xu is a research engineer at Palo Alto Networks in CA, United States. He joined Palo Alto Networks in 2014 and worked in the area of Internet security. He earned his Ph.D. degree at Texas A&M University, College Station in 2014. His research interests include web security, malware analysis, detection and system security.
	Tongbo Luo Tongbo Luo is a principle security researcher at Palo Alto Networks. His current research interests include cybersecurity, mobile security and security data analysis. He obtained his M.S. and Ph.D. degrees in computer science from Syracuse University in 2014. He is active in mobile security, cybersecurity, IoT security and applied machine learning for security problems.
	Wei Xu Wei Xu is a security researcher at Palo Alto Networks. His current research interests include web security, network security and security data analysis. His past research works have been published in both academic and industry circles. He was a speaker at VB2012/2014/2015 and Blackhat 2013. He received his B.S and M.S. degrees in electrical engineering from Tsinghua University, Beijing,China, in 2005 and 2007 respectively. He obtained his Ph.D degree in computer science from Penn State University in 2013.
	Kyle Sanders Kyle Sanders has worked in the IT industry for the last 11 years and is currently the team lead for malware research at Palo Alto Networks. His research interests are in automated malware detection, network forensics and code analysis.
	Xin Ouyang Xin Ouyang is a senior manager at Palo Alto Networks. His current research interests include intrusion detection and prevention systems, web security, security data analysis, and security of the Internet of Things.