Aprendizado de máquina para previsão do tempo de execução de aplicações Spark

Abstract

The rise of big data applications brought along a series of difficult challenges regarding the allocation of hardware and software resources. Typically these applications are known for being computationally expensive and having high heterogeneity on how they operate, making the task of estimating application’s execution time very challenging. It may be still possible to correlate features extracted from the cloud environment and from the input dataset to the execution time. Such relationship may then be used to predict execution times. Based on such assumption, this work explores machine learning (ML) models to the task of predict execution time of Spark applications. This work investigates four ML models as well as different features, while also comparing their results against the current state-of-the-art. All models are evaluated in several scenarios and configurations, producing results that are significantly superior to the state-of-the-art in various cases.