Extensão de um ambiente de computação de alto desempenho para o processamento de dados massivos

Abstract

High performance computing (HPC) and mass data processing (big data) are two trends in computing systems for dealing with complex or large data problems. Each of these systems specializes in a set of specific problems with unique approaches, however, currently such systems are beginning to converge, often brought on by the mixing of domains of a given problem. An example of this is the Superscalar COMP (COMPSs), a parallel and distributed programming model originating from the HPC world that has been integrated into new functionalities usually related to big data environments. This paper presents our contribution on this convergence path in order to process massive data by integrating COMPS into HDFS, one of the most widely used distributed file systems in big data, and Lemonade, a data mining and analysis tool developed at Universidade Federal de Minas Gerais (UFMG). The results show that integration with HDFS benefits the COMPS not only by data abstraction, which simplifies access to data, but also by increased performance in executions that need to read large volumes of data, caused by the reorganization of data transfer by network. In addition, Lemonade integration facilitates its use and popularization in the area of Data Science by providing good implementations of algorithms and operations for data domain specialists who wish to develop and run COMPS applications with a higher level of abstraction.