Um ambiente de programação para sistemas distribuídos com grande volume de dados

Abstract

Anthill is an implementation of a parallel programming environment, based on filter-stream model. Filter-stream is an efficient model for processing large ammounts of data and works as a network pipeline, where each stage processes data and transmits the result for the next one. Stages heavily loaded can be replicated, balancing the load among the pipeline components and having a constant data flow. In this work, the features of the programming model and the environment are presented: filters and transparent copies, data parallelism interface, stream communication, distributed state through labeled streams and system usage. Results of two real applications that make use of our system will also be presented, showing. Anthill can reach superlinear speedups. Anthill implementation is based on DataCutter, a system developed by University of Maryland and the Ohio State University. The addition of new features, like labeled stream and work division interface, makes it possible for a greater range of applications to be run in the environment.