A Polymorphic Shellcode Detection Mechanism in the Network

Hsiang-Lun Huang; Tzong-Jye Liu; Kuong-Ho Chen; Chyi-Ren Dow; Lih-Chyau Wuu

2nd International ICST Conference on Scalable Information Systems

Research Article

A Polymorphic Shellcode Detection Mechanism in the Network

Download1699 downloads

Cite: BibTeX Plain Text

@INPROCEEDINGS{10.4108/infoscale.2007.225,
    author={Hsiang-Lun Huang and Tzong-Jye Liu and Kuong-Ho Chen and Chyi-Ren Dow and Lih-Chyau Wuu},
    title={A Polymorphic Shellcode Detection Mechanism in the Network},
    proceedings={2nd International ICST Conference on Scalable Information Systems},
    proceedings_a={INFOSCALE},
    year={2010},
    month={5},
    keywords={Buffer overflow intrusion detection system polymorphic shellcode.},
    doi={10.4108/infoscale.2007.225}
}

Hsiang-Lun Huang
Tzong-Jye Liu
Kuong-Ho Chen
Chyi-Ren Dow
Lih-Chyau Wuu
Year: 2010
A Polymorphic Shellcode Detection Mechanism in the Network
INFOSCALE
ICST
DOI: 10.4108/infoscale.2007.225

Hsiang-Lun Huang¹^,*, Tzong-Jye Liu¹^,*, Kuong-Ho Chen¹^,*, Chyi-Ren Dow¹^,*, Lih-Chyau Wuu²^,*

1: Department of Information Engineering and Computer Science Feng Chia University Taichung, Taiwan, R.O.C.
2: Institute of Computer Science and Information Engineering National Yunlin University of Science and Technology Yunlin, Taiwan, R.O.C.

*Contact email: m9405100@fcu.edu.tw, tjliu@fcu.edu.tw, cyne@pluto.iecs.fcu.edu.tw, crdow@fcu.edu.tw, wuulc@yuntech.edu.tw

Abstract

Buffer overflow attack is a major security problem in recent years. The polymorphism technique for shellcode becomes more and more popular along with development of Internet. This paper proposes a method to detect the polymorphic shellcode for Windows operating system. The proposed approach relies on an IA-32 CPU emulator that executes instruction sequences and analyze the behavior of polymorphic shellcode. The experimental results show that the approach is able to detect polymorphic shellcode accurately.

Keywords: Buffer overflow intrusion detection system polymorphic shellcode.

Published: 2010-05-16
Modified: 2011-09-11

: http://dx.doi.org/10.4108/infoscale.2007.225