Q. Zhang, D. S. Reeves, P. Ning, and S. P. Iyer, "Analyzing Network Traffic
to Detect Self-Decrypting Exploit Code", in Proc. of the ACM Symposium on
Information, Computer, and Communications Security (ASIACCS 2007), March
Remotely-launched software exploits are a common way for
attackers to intrude into vulnerable computer systems. As
detection techniques improve, remote exploitation techniques
are also evolving. Recent techniques for evasion of exploit
detection include polymorphism (code encryption) and metamorphism
(code obfuscation). This paper addresses the
problem of detecting in network traffic polymorphic remote
exploits that are encrypted, and that self-decrypt before
launching the intrusion. Such exploits pose a great challenge
to existing malware detection techniques, partly due
to the non-obvious starting location of the exploit code in
the network payload.
We describe a new method for detecting self-decrypting
exploit codes. This method scans network traffic for the
presence of a decryption routine, which is characteristic of
such exploits. The proposed method uses static analysis and
emulated instruction execution techniques. This improves
the accuracy of determining the starting location and instructions
of the decryption routine, even if self-modifying
code is used. The method outperforms approaches that have
been previously proposed, both in terms of detection capabilities,
and in detection accuracy.
The proposed method has been implemented and tested
on current polymorphic exploits, including ones generated
by state-of-the-art polymorphic engines. All exploits have
been detected (i.e., a 100% detection rate), including those
for which the decryption routine is dynamically coded, or
self-modifying. The false positive rate is close to 0%. Running
time is approximately linear in the size of the network
payload being analyzed.
Q. Zhang and D. S. Reeves,
"MetaAware: Identifying Metamorphic Malware",
in Proc. of the 23rd Annual Computer Security Applications
Conference (ACSAC 2007), December 2007, IEEE Computer Society, pp.
Detection of malicious software (malware) by the use of static signatures is
often criticized for being overly simplistic. Available methods of
obfuscating code (so-called metamorphic malware) will invalidate the use of
a fixed signature, without changing the harmful effects of the software.
This paper presents a new approach for recognizing meta morphic malware. The
method uses fully automated static analysis of executables to summarize and
compare program semantics, based primarily on the pattern of library or
system functions which are called. The proposed method has been prototyped
and evaluated using randomized benchmark programs, instances of known
malware program variants, and utility software available in multiple
releases. The results demonstrate three important capabilities of the
proposed method: (a) it does well at identifying metamorphic variants of
common malware; (b) it distinguishes easily between programs that are not
related; and, (c) it can identify and detect program variations, or code
reuse. Such variations can be due to insertion of malware (such as viruses)
into the executable of a host program. We argue that this method of
metamorphic code detection will be difficult for malware writers to bypass.
Y. Park and D. S. Reeves, "Identification of Bot Commands
by Run-Time Execution Monitoring", in
Proc. of the 25th Annual Computer Security Applications Conference
(ACSAC 2009), December 20009, IEEE Computer Society, pp. 321-330.
Botnets pose serious threats to the Internet. In spite of
substantial efforts to address the issue, botnets are dramatically
spreading. Bots in a botnet execute commands under the control of
the botnet owner or controller. A first step in protecting against
botnets is identification of their presence, and activities. In this
paper, we propose a method of identifying the high-level commands
executed by bots. The method uses run- time monitoring of bot
execution to capture and analyze run- time call behavior. We find
that bots have distinct behavior patterns when they perform
pre-programmed bot commands. The patterns are characterized by
sequences of common API calls at regular intervals. We demonstrate
that commands aiming to achieve the same result have very similar
API call behavior in bot variants, even when they are from different
bot families. We implemented and evaluated a prototype of our
method. Run-time monitoring is accomplished by user-level hooking.
In the experiments, the proposed method successfully identified the
bot commands being executed with a success rate of 97%. The ability
of the method to identify bot commands despite the use of execution
obfuscation is also addressed.
Y. Park and D. S. Reeves, "Fast Malware Classification by Automated
Behavioral Graph Matching", in
Proc. of the Sixth Annual Workshop on Cyber Security and Information
Intelligence Research (CSIIRW '10),
April, 2010, ACM.
Malicious software (malware) is a serious problem in the Internet.
Malware classification is useful for detection and analysis of new
threats for which signatures are not available, or possible (due to
polymorphism). This paper proposes a new malware classification
method based on maximal common subgraph detection. A behavior graph
is obtained by capturing system calls during the execution (in a
sandboxed environment) of the suspicious software. The method has
been implemented and tested on a set of 300 malware instances in 6
families. Results demonstrate the method effectively groups the
malware instances, compared with previous methods of classification,
is fast, and has a low false positive rate when presented with
Y. Park, Q. Zhang, D. S. Reeves, and V. Mulukutla,
"AntiBot: Clustering Common Semantic Patterns for Bot Detection",
Proc. of 10th IEEE/IPSJ International Symposium on Applications and the Internet
( COMPSAC / SAINT 2010), July 2010, IEEE Computer Society.
[ Publisher's website]
Among malicious software (malware), autonomous malicious programs, called bots, are a serious problem in the Internet. The bot writers have developed a variety of techniques to evade simple signature-based detection. Concise representations of malware behavior, or semantic patterns, are much harder to evade or obfuscate. However, generating a semantic pattern for every program instance is time-consuming, and comparing with a large number of patterns creates a challenge for timely identification of bots. This paper proposes an automated approach to generate semantic patterns for bot detection. Unlike previous approaches, it is intended to find one pattern that accurately represents the important behavior of an entire class of bots, rather than of individual instances. Doing so has advantages for fast malware identification, and for distinguishing new classes of attacks from previously-seen attacks. The work uses static analysis to characterize bot behaviors, and proposes to use hierarchical clustering of the resulting semantic patterns from a set of bot programs. The goal is to identify critical, common semantic behavior that represents the functions of an entire class of the malware. This method has been prototyped and evaluated on real-world malicious bot software. Depending on parameter choices, our approach can achieve more than 95% detection rates and less than 5% false positive rates on a large set of bot programs and non-bot executables.
Y. Park and D. S. Reeves, "Deriving Common Malware Behavior through Graph Clustering", in Proc. of the 6th ACM Symposium on Information, Computer, and
Communications Security (ASIACCS2011),
March 2011, ACM, pp. 497-502.
Detection of malicious software (malware) continues to be a problem as hackers devise new ways to evade available methods. The proliferation of malware and malware variants requires methods that are both powerful, and fast to execute. This paper proposes a method to derive the common execution behavior of a family of malware instances. For each instance, a graph is constructed that represents kernel objects and their attributes, based on system call traces. The method combines these graphs to develop a supergraph for the family. This supergraph contains a subgraph, called the HotPath, which is observed during the execution of all the malware instances. The proposed method is scalable, identifies previously-unseen malware instances, shows high malware detection rates, and false positive rates close to 0%.