Spark Memory Management

Wei Zhang; Jingmei Li

Advanced Hybrid Information Processing. First International Conference, ADHIP 2017, Harbin, China, July 17–18, 2017, Proceedings

Research Article

Spark Memory Management

Download

245 downloads

Cite: BibTeX Plain Text

@INPROCEEDINGS{10.1007/978-3-319-73317-3_9,
    author={Wei Zhang and Jingmei Li},
    title={Spark Memory Management},
    proceedings={Advanced Hybrid Information Processing. First International Conference, ADHIP 2017, Harbin, China, July 17--18, 2017, Proceedings},
    proceedings_a={ADHIP},
    year={2018},
    month={2},
    keywords={Spark framework Memory management Memory overflow},
    doi={10.1007/978-3-319-73317-3_9}
}

Wei Zhang
Jingmei Li
Year: 2018
Spark Memory Management
ADHIP
Springer
DOI: 10.1007/978-3-319-73317-3_9

Wei Zhang¹^,*, Jingmei Li¹

1: Harbin Engineering University

*Contact email: zhangwei72@hrbeu.edu.cn

Abstract

In order to obtain detailed information about Spark framework and realize fine grained monitoring of cluster operation information, a performance analysis system is designed. Therefore, the problems of Spark1.6 memory management scheme are researched in depth and improved. The experimental results show that the original memory management scheme is inconsistent with the requirements of Spark’s official website. However, the improved memory management scheme not only meets the requirements of Spark’s official website, but also makes the application run successfully under the condition of small memory capacity.

Keywords: Spark framework Memory management Memory overflow

Published: 2018-02-09
Appears in: SpringerLink

: http://dx.doi.org/10.1007/978-3-319-73317-3_9