funded by

Home · Papers · Participants · Software · Theses



  1. Q. Zhang, D. S. Reeves, P. Ning, and S. P. Iyer, "Analyzing Network Traffic to Detect Self-Decrypting Exploit Code", in Proc. of the ACM Symposium on Information, Computer, and Communications Security (ASIACCS 2007), March 2007, Singapore. [Publisher's website] [PDF]

    Remotely-launched software exploits are a common way for attackers to intrude into vulnerable computer systems. As detection techniques improve, remote exploitation techniques are also evolving. Recent techniques for evasion of exploit detection include polymorphism (code encryption) and metamorphism (code obfuscation). This paper addresses the problem of detecting in network traffic polymorphic remote exploits that are encrypted, and that self-decrypt before launching the intrusion. Such exploits pose a great challenge to existing malware detection techniques, partly due to the non-obvious starting location of the exploit code in the network payload.
    We describe a new method for detecting self-decrypting exploit codes. This method scans network traffic for the presence of a decryption routine, which is characteristic of such exploits. The proposed method uses static analysis and emulated instruction execution techniques. This improves the accuracy of determining the starting location and instructions of the decryption routine, even if self-modifying code is used. The method outperforms approaches that have been previously proposed, both in terms of detection capabilities, and in detection accuracy.
    The proposed method has been implemented and tested on current polymorphic exploits, including ones generated by state-of-the-art polymorphic engines. All exploits have been detected (i.e., a 100% detection rate), including those for which the decryption routine is dynamically coded, or self-modifying. The false positive rate is close to 0%. Running time is approximately linear in the size of the network payload being analyzed.
  2. Q. Zhang and D. S. Reeves, "MetaAware: Identifying Metamorphic Malware", in Proc. of the 23rd Annual Computer Security Applications Conference (ACSAC 2007), December 2007, IEEE Computer Society, pp. 411-420. [Publisher's website] [PDF]

    Detection of malicious software (malware) by the use of static signatures is often criticized for being overly simplistic. Available methods of obfuscating code (so-called metamorphic malware) will invalidate the use of a fixed signature, without changing the harmful effects of the software. This paper presents a new approach for recognizing meta morphic malware. The method uses fully automated static analysis of executables to summarize and compare program semantics, based primarily on the pattern of library or system functions which are called. The proposed method has been prototyped and evaluated using randomized benchmark programs, instances of known malware program variants, and utility software available in multiple releases. The results demonstrate three important capabilities of the proposed method: (a) it does well at identifying metamorphic variants of common malware; (b) it distinguishes easily between programs that are not related; and, (c) it can identify and detect program variations, or code reuse. Such variations can be due to insertion of malware (such as viruses) into the executable of a host program. We argue that this method of metamorphic code detection will be difficult for malware writers to bypass.

  3. Y. Park and D. S. Reeves, "Identification of Bot Commands by Run-Time Execution Monitoring", in Proc. of the 25th Annual Computer Security Applications Conference (ACSAC 2009), December 20009, IEEE Computer Society, pp. 321-330. [Publisher's website] [PDF]

    Botnets pose serious threats to the Internet. In spite of substantial efforts to address the issue, botnets are dramatically spreading. Bots in a botnet execute commands under the control of the botnet owner or controller. A first step in protecting against botnets is identification of their presence, and activities. In this paper, we propose a method of identifying the high-level commands executed by bots. The method uses run- time monitoring of bot execution to capture and analyze run- time call behavior. We find that bots have distinct behavior patterns when they perform pre-programmed bot commands. The patterns are characterized by sequences of common API calls at regular intervals. We demonstrate that commands aiming to achieve the same result have very similar API call behavior in bot variants, even when they are from different bot families. We implemented and evaluated a prototype of our method. Run-time monitoring is accomplished by user-level hooking. In the experiments, the proposed method successfully identified the bot commands being executed with a success rate of 97%. The ability of the method to identify bot commands despite the use of execution obfuscation is also addressed.

  4. Y. Park and D. S. Reeves, "Fast Malware Classification by Automated Behavioral Graph Matching", in Proc. of the Sixth Annual Workshop on Cyber Security and Information Intelligence Research (CSIIRW '10), April, 2010, ACM. [Publisher's website] [PDF]

    Malicious software (malware) is a serious problem in the Internet. Malware classification is useful for detection and analysis of new threats for which signatures are not available, or possible (due to polymorphism). This paper proposes a new malware classification method based on maximal common subgraph detection. A behavior graph is obtained by capturing system calls during the execution (in a sandboxed environment) of the suspicious software. The method has been implemented and tested on a set of 300 malware instances in 6 families. Results demonstrate the method effectively groups the malware instances, compared with previous methods of classification, is fast, and has a low false positive rate when presented with benign software.

  5. Y. Park, Q. Zhang, D. S. Reeves, and V. Mulukutla, "AntiBot: Clustering Common Semantic Patterns for Bot Detection", Proc. of 10th IEEE/IPSJ International Symposium on Applications and the Internet ( COMPSAC / SAINT 2010), July 2010, IEEE Computer Society. [ Publisher's website] [PDF]

    Among malicious software (malware), autonomous malicious programs, called bots, are a serious problem in the Internet. The bot writers have developed a variety of techniques to evade simple signature-based detection. Concise representations of malware behavior, or semantic patterns, are much harder to evade or obfuscate. However, generating a semantic pattern for every program instance is time-consuming, and comparing with a large number of patterns creates a challenge for timely identification of bots. This paper proposes an automated approach to generate semantic patterns for bot detection. Unlike previous approaches, it is intended to find one pattern that accurately represents the important behavior of an entire class of bots, rather than of individual instances. Doing so has advantages for fast malware identification, and for distinguishing new classes of attacks from previously-seen attacks. The work uses static analysis to characterize bot behaviors, and proposes to use hierarchical clustering of the resulting semantic patterns from a set of bot programs. The goal is to identify critical, common semantic behavior that represents the functions of an entire class of the malware. This method has been prototyped and evaluated on real-world malicious bot software. Depending on parameter choices, our approach can achieve more than 95% detection rates and less than 5% false positive rates on a large set of bot programs and non-bot executables.

  6. Y. Park and D. S. Reeves, "Deriving Common Malware Behavior through Graph Clustering", in Proc. of the 6th ACM Symposium on Information, Computer, and Communications Security (ASIACCS2011), March 2011, ACM, pp. 497-502. [Publisher's website] [PDF]

    Detection of malicious software (malware) continues to be a problem as hackers devise new ways to evade available methods. The proliferation of malware and malware variants requires methods that are both powerful, and fast to execute. This paper proposes a method to derive the common execution behavior of a family of malware instances. For each instance, a graph is constructed that represents kernel objects and their attributes, based on system call traces. The method combines these graphs to develop a supergraph for the family. This supergraph contains a subgraph, called the HotPath, which is observed during the execution of all the malware instances. The proposed method is scalable, identifies previously-unseen malware instances, shows high malware detection rates, and false positive rates close to 0%.


Douglas S. Reeves · Department of Computer Science · NC State University

Last modified on Thursday, 04-Oct-2012 14:20:38 EDT
Send comments to: web page maintainer
designed with