2nd International ICST Conference on Scalable Information Systems

Research Article

A Polymorphic Shellcode Detection Mechanism in the Network

Download1653 downloads
  • @INPROCEEDINGS{10.4108/infoscale.2007.225,
        author={Hsiang-Lun Huang and Tzong-Jye Liu and Kuong-Ho Chen and Chyi-Ren Dow and Lih-Chyau Wuu},
        title={A Polymorphic Shellcode Detection Mechanism in the Network},
        proceedings={2nd International ICST Conference on Scalable Information Systems},
        proceedings_a={INFOSCALE},
        year={2010},
        month={5},
        keywords={Buffer overflow intrusion detection system polymorphic shellcode.},
        doi={10.4108/infoscale.2007.225}
    }
    
  • Hsiang-Lun Huang
    Tzong-Jye Liu
    Kuong-Ho Chen
    Chyi-Ren Dow
    Lih-Chyau Wuu
    Year: 2010
    A Polymorphic Shellcode Detection Mechanism in the Network
    INFOSCALE
    ICST
    DOI: 10.4108/infoscale.2007.225
Hsiang-Lun Huang1,*, Tzong-Jye Liu1,*, Kuong-Ho Chen1,*, Chyi-Ren Dow1,*, Lih-Chyau Wuu2,*
  • 1: Department of Information Engineering and Computer Science Feng Chia University Taichung, Taiwan, R.O.C.
  • 2: Institute of Computer Science and Information Engineering National Yunlin University of Science and Technology Yunlin, Taiwan, R.O.C.
*Contact email: m9405100@fcu.edu.tw, tjliu@fcu.edu.tw, cyne@pluto.iecs.fcu.edu.tw, crdow@fcu.edu.tw, wuulc@yuntech.edu.tw

Abstract

Buffer overflow attack is a major security problem in recent years. The polymorphism technique for shellcode becomes more and more popular along with development of Internet. This paper proposes a method to detect the polymorphic shellcode for Windows operating system. The proposed approach relies on an IA-32 CPU emulator that executes instruction sequences and analyze the behavior of polymorphic shellcode. The experimental results show that the approach is able to detect polymorphic shellcode accurately.