TransNet: Unseen Malware Variants Detection Using Deep Transfer Learning

The ever-increasing amount and variety of malware on the Internet have presented significant challenges to the interconnected network community. The emergence of unseen malware variants has resulted in a different distribution of features and labels in the training and testing datasets. For widely used machine learning-based detection methods, the issue of dataset shift will render the trained model ineffective in the face of new data. However, it is a laborious and tedious undertaking whether relearning features to describe new data or collecting large amounts of labeled samples to retrain the classifiers. To address these problems, this paper proposes TransNet, a framework based on deep transfer learning for unseen malware variants detection. We first convert the raw traffic represented by sessions containing data from all layers of the OSI model into fixed-size RGB images through data preprocessing. Afterward, based on the ResNet-50 model pre-trained on the ImageNet, we replace Batch Normalization with Transferable Normalization as the normalization layer to construct our deep transfer learning model. In this way, our approach leverages deep learning to avoid the problem of traditional machine learning in relying on expert knowledge and uses transfer learning to address the issue of domain shift. We test the effectiveness of different methods with a thorough set of experiments. TransNet achieves 95.89% accuracy and 96.09% F-measure on two public datasets from the real-world environment, which is higher than comparative methods. Meantime, our method ranks first on all ten subtasks, showing that it can detect unseen malware variants with stable and excellent performance. Moreover, the distribution discrepancy computed by our method is much smaller than other approaches, which illustrates that our method successfully reduces the shift of data distributions.