Research Article
ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations
@INPROCEEDINGS{10.1007/978-3-030-92511-6_5, author={Stefan Herrnleben and Maximilian Leidinger and Veronika Lesch and Thomas Prantl and Johannes Grohmann and Christian Krupitzer and Samuel Kounev}, title={ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations}, proceedings={Performance Evaluation Methodologies and Tools. 14th EAI International Conference, VALUETOOLS 2021, Virtual Event, October 30--31, 2021, Proceedings}, proceedings_a={VALUETOOLS}, year={2021}, month={12}, keywords={IoT Publish/subscribe Benchmarking Load testing}, doi={10.1007/978-3-030-92511-6_5} }- Stefan Herrnleben
Maximilian Leidinger
Veronika Lesch
Thomas Prantl
Johannes Grohmann
Christian Krupitzer
Samuel Kounev
Year: 2021
ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations
VALUETOOLS
Springer
DOI: 10.1007/978-3-030-92511-6_5
Abstract
Efficient and dependable communication is a highly relevant aspect for Internet of Things (IoT) systems in which tiny sensors, actuators, wearables, or other smart devices exchange messages. Various publish/subscribe protocols address the challenges of communication in IoT systems. The selection process of a suitable protocol should consider the communication behavior of the application, the protocol’s performance, the resource requirements on the end device, and the network connection quality, as IoT environments often rely on wireless networks. Benchmarking is a common approach to evaluate and compare systems, considering the performance and aspects like dependability or security. In this paper, we present our IoT communication benchmarking frameworkComBenchfor publish/subscribe protocols focusing on constrained networks with varying quality conditions. The benchmarking framework supports system designers, software engineers, and application developers to select and investigate the behavior of communication protocols. Our benchmarking framework contributes to (i) show the impact of fluctuating network quality on communication, (ii) compare multiple protocols, protocol features, and protocol implementations, and (iii) analyze scalability, robustness, and dependability of clients, networks, and brokers in different scenarios. Our case study demonstrates the applicability of our framework to support the decision for the best-suited protocol in various scenarios.
