Escalonamento multiplataforma de fluxos de processamento em ciência de dados

Abstract

Cross-platform data processing systems are tools that can lead to efficient execution of complex workflows across heterogeneous processing tools. However, existing solutions face critical limitations in real-world deployments: (i) current interfaces rely on simplified operator abstractions such as map/filter/reduce that aggregate diverse high-level operations as equivalent, ignoring crucial computational distinctions and data characteristics; (ii) furthermore, the lack of prior statistics regarding input data, combined with the dynamic evolution of data properties throughout workflow execution, leads to suboptimal scheduling decisions, as existing schedulers fail to account for data volume and schema changes across pipeline stages. In this context, we propose a comprehensive framework for cross-platform scheduling and execution of Data Science tasks utilizing DataFrame, a modern data abstraction. Our approach addresses these limitations through an extended graph-based modeling of workflows. This enhanced graph structure explicitly captures: (i) the evolution of data volumes and schema characteristics across pipeline stages, (ii) heterogeneous execution costs across available tools, and (iii) data migration overheads between distinct tools. The proposed methodology integrates three key technical contributions: (i) a rule-based transformation system for workflow optimization that operates on the extended graph representation; (ii) machine learning models for accurate estimation of execution times; and (iii) a dynamic recalibration mechanism that leverages historical execution data to adapt to evolving workloads. We implemented our model as an extension of an existing visual programming environment for Data Science, demonstrating practical viability across three distinct hardware configurations and multiple processing backends. Experimental validation demonstrates that our approach achieves 93% accuracy in selecting the most suitable processing tool and 75% accuracy in identifying optimal hardware configurations. Furthermore, the system enables performance improvements of up to 3.3 times compared to traditional single-tool execution by scheduling tasks across multiple tools. The solution also exhibits robust stability across diverse execution environments and maintains the capability to evolve through continuous learning from historical executions.