MapReduce Distributed Computing System on A Serverless Platform

As serverless entering clouds computing, MapReduce, a traditional design of massive data processing, is slowly becoming a part of serverless computing engines. This study formulates and deploys a lightweight, elastic, and pay-as-you-go MapReduce distributed computing platform on the open-source serverless platform, OpenWhisk. The model breaks down the classical MapReduce model into serverless functions and relies on an object storage service (MinIO) to accomplish data interchange among functions, which can be dynamic in resource scheduling and task coordination. The findings of this experiment indicate that the system exhibits the sharing of good performance in parallelism and cost-effectiveness in small and medium-scale data processing processes, particularly when they lead to short-term, high-concurrent processing and intermittent computing processes. But when the data processing involves a large amount of data, storage and network communication overheads are the primary bottlenecks. The work has practical references in designing and optimizing distributed computing in a serverless environment.