Runtime Performance Prediction of Big Data Workflows with I/O-aware Simulation

Modelling and simulation of Big Data analytics processes running in the cloud is a difficult problem which introduces many challenges. The major one is the collection of training data which is scarce and costly to obtain, due to large scale and long runtime of those processes. In our previous work, we proposed a methodology to model, simulate and predict the runtime of Big Data processes such as complex Next Generation Sequencing (NGS) pipelines. The major contribution of our simulation methodology is that it provides a reasonable prediction of runtime for testing data much larger than the training inputs. To further improve the accuracy of prediction we present now an extension of our previous work that can model cloud data storage. Our simulation framework is based on CloudSim and WorkflowSim, to which we have added a shared storage component. We present the design and implementation of the storage extension together with an evaluation performed on selected scientific workflows: the Pegasus Montage workflow and NGS pipeline implemented in e-Science Central. The evaluation shows that the proposed extension works correctly and can improve prediction accuracy for our largest 390GB input dataset by about 16% when compared to previous results.