Research Article
NetFlow Datasets for Machine Learning-Based Network Intrusion Detection Systems
@INPROCEEDINGS{10.1007/978-3-030-72802-1_9, author={Mohanad Sarhan and Siamak Layeghy and Nour Moustafa and Marius Portmann}, title={NetFlow Datasets for Machine Learning-Based Network Intrusion Detection Systems}, proceedings={Big Data Technologies and Applications. 10th EAI International Conference, BDTA 2020, and 13th EAI International Conference on Wireless Internet, WiCON 2020, Virtual Event, December 11, 2020, Proceedings}, proceedings_a={BDTA \& WICON}, year={2021}, month={7}, keywords={Network intrusion detection system NetFlow Machine learning Network datasets Network features}, doi={10.1007/978-3-030-72802-1_9} }- Mohanad Sarhan
Siamak Layeghy
Nour Moustafa
Marius Portmann
Year: 2021
NetFlow Datasets for Machine Learning-Based Network Intrusion Detection Systems
BDTA & WICON
Springer
DOI: 10.1007/978-3-030-72802-1_9
Abstract
Machine Learning (ML)-based Network Intrusion Detection Systems (NIDSs) have become a promising tool to protect networks against cyberattacks. A wide range of datasets are publicly available and have been used for the development and evaluation of a large number of ML-based NIDS in the research community. However, since these NIDS datasets have very different feature sets, it is currently very difficult to reliably compare ML models across different datasets, and hence if they generalise to different network environments and attack scenarios. The limited ability to evaluate ML-based NIDSs has led to a gap between the extensive academic research conducted and the actual practical deployments in the real-world networks. This paper addresses this limitation, by providing five NIDS datasets with a common, practically relevant feature set, based on NetFlow. These datasets are generated from the following four existing benchmark NIDS datasets: UNSW-NB15, BoT-IoT, ToN-IoT, and CSE-CIC-IDS2018. We have used the raw packet capture files of these datasets, and converted them to the NetFlow format, with a common feature set. The benefits of using NetFlow as a common format include its practical relevance, its wide deployment in production networks, and its scaling properties. The generated NetFlow datasets presented in this paper have been labelled for both binary- and multi-class traffic and attack classification experiments, and we have made them available for to the research community []. As a use-case and application scenario, the paper presents an evaluation of an Extra Trees ensemble classifier across these datasets.
