Research Article
A Machine Learning-Based Elastic Strategy for Operator Parallelism in a Big Data Stream Computing System
@INPROCEEDINGS{10.1007/978-3-030-93479-8_1, author={Wei Li and Dawei Sun and Shang Gao and Rajkumar Buyya}, title={A Machine Learning-Based Elastic Strategy for Operator Parallelism in a Big Data Stream Computing System}, proceedings={Broadband Communications, Networks, and Systems. 12th EAI International Conference, BROADNETS 2021, Virtual Event, October 28--29, 2021, Proceedings}, proceedings_a={BROADNETS}, year={2022}, month={1}, keywords={Operator parallelism Runtime awareness Resource allocation Machine learning Stream computing Distributed system}, doi={10.1007/978-3-030-93479-8_1} }- Wei Li
Dawei Sun
Shang Gao
Rajkumar Buyya
Year: 2022
A Machine Learning-Based Elastic Strategy for Operator Parallelism in a Big Data Stream Computing System
BROADNETS
Springer
DOI: 10.1007/978-3-030-93479-8_1
Abstract
Elastic scaling in/out of operator parallelism degree is needed for processing real time dynamic data streams under low latency and high stability requirements. Usually the operator parallelism degree is set when a streaming application is submitted to a stream computing system and kept intact during runtime. This may substantially affect the performance of the system due to the fluctuation of input streams and availability of system resources. To address the problems brought by the static parallelism setting, we propose and implement a machine learning based elastic strategy for operator parallelism (named Me-Stream) in big data stream computing systems. The architecture of Me-Stream and its key models are introduced, including parallel bottleneck identification, parameter plan generation, parameter migration and conversion, and instances scheduling. Metrics of execution latency and process latency of the proposed scheduling strategy are evaluated on the widely used big data stream computing system Apache Storm. The experimental results demonstrate the efficiency and effectiveness of the proposed strategy.
