PTFS - Previsão e Tratamento de Falhas em modelos de Stacking

Abstract

Stacking models are widely used in Automatic Text Classification (ATC). Stacking combines different classifiers (base models) through a meta-classifier that exploits the different abilities and complementarities of the base models to leverage effectiveness in classification. However, in traditional stacking approaches, the meta-classifier is limited to learning only a single combination of base models for all instances in a database. This limitation hampers the performance of stacking because, when a base model fails (or has a high chance of failing) to correctly classify a document (test), its importance for the final decision of the meta-classifier should be reduced. However, this does not occur in current solutions. In this work, we focus on addressing this limitation of the meta-classifier in dealing with potential failures of base models. More specifically, we are interested in developing strategies to assist the meta-classifier in adjusting the importance of the base models according to the expected success of each model for each specific document. Our work aims to answer research questions such as: (i) Is it possible to achieve the maximum potential of stacking by predicting and addressing failures of base models? (ii) Is it possible to find the most suitable base model to classify a specific document? To answer such questions, we present framework Prediction and Treatment of failures in Stacking models (PTFS). Our framework encompasses three strategies (Error Detection, Best Model, and Hard Docs) aimed at identifying and addressing failures of base models, potentially reducing the impact of these models' failures on the performance of the final meta-classifier and stacking overall. We present a comprehensive evaluation of the strategies contained in PTFS using multiple databases and various base classifiers (models). Despite negative answers to several research questions, our study provides interesting insights and analyses that can guide future work.