Conception of a load balancing strategy for CLOAK-Reduce

Distributed systems are highly heterogeneous, dynamic and unstable. It is therefore realistic to expect that some resources will fail during use. To overcome these problems and achieve better performance, it is necessary to implement load balancing algorithms that are adapted to any situation where some nodes are overloaded while others are less so or are even idle. Load balancing between JobManager and JobManagers candidates, and between JobManagers of the same scheduler or load balancing between Schedulers, implies that additional loads are only done hierarchically. In this paper, we propose a two-level dynamic, hierarchical and decentralised load balancing strategy focusing on three performance indicators namely: response time, process latency and running time of MapReduce jobs. The first level of load balancing is intra-scheduler, in order to avoid the use of the large-scale communication network, and the second level of load balancing is inter-scheduler, for load regulation of our whole system. The proposed solution provides a better optimisation of the load balancing process and an improvement of the task mean response time with minimal communication.