A robust deep convolutional neural network model for text categorization

Abstract

Text categorization is the task of automatically assigning pre-defined categories to documents written in natural languages and it is one of the most important tasks in Natural Language Processing (NLP) domain applications. Traditional machine learning techniques rely on handcrafted features such as ngrams, negation words, punctuation, emoticons, stop words, elongated words and lexicons to build their models. This approach, called feature engineering, in addition to being labor intensive, results in models that, in general, present poor performance on tasks for what they have not been specifically tailored.In this work, we propose a robust deep learning Convolutional Neural Network (CNN) model named Deep NLP. Our model adopts a compositional approach, in which the design of the deep CNN architecture induces the creation of a hierarchical representation for thetext, through the extraction of intermediate representations for the words and sentences of the text. The initial word representations, called word embeddings, are obtained from a pretrained unsupervised neural language model and they are adjusted for the context of the taskthat the model is being trained. We evaluated our model comparing its accuracy against the results reported by some traditional and deep learning models in text categorization tasks using six large-scale datasets. The results show that our model is robust in the sense that, even when we use the same hyperparameters, it surpasses the accuracy of the state-of-the-art models in different text categorization tasks. The results also show that the use of a semantic synonyms dictionarytogether with the word embeddings helps to generalize the representations learned by the model increasing its accuracy.