Sumarização de texto em Deep Learning como etapa inicial para a construção de um modelo de recuperação da informação: análise do setor de mineração no Brasil

Abstract

At the turn of the 20th century to the 21st, the world experienced a paradigm shift, moving from a state of data scarcity to a state of data overabundance. This new scenario generated Big Data, a series of tools and technologies for processing and storing data. Linked to it is Artificial Intelligence, which gains more relevance by providing new methods for dealing with huge silos of data, including textual ones, through the use of Natural Language Processing. Although Natural Language Processing has several tasks, Automatic Text Summary stands out for reducing the amount of writing without losing meaning, providing the application of other Natural Language Processing tasks and Machine Learning algorithms. Given this scenario, there is a need to retrieve and analyze information that shows structural changes in the macroenvironment of the mining sector in Brazil. To this end, around 3,224 news items from websites and newspapers about this sector were collected, pre-processed and summarized. After being summarized, the news went through steps of measuring similarity, measuring the degree of sentiment and clustering the summaries. The summaries, in turn, were grouped into General and No Semantic Similarity contexts, and for each year of the No Semantic Similarity context, Clusters were also generated for analysis. Next, a dataframe was built, with the final output, and a data panel. Then, the constructed method was evaluated by 15 experts, 4 from Mining and 11 from Data and Information, of which 2 had specialization in both areas. In terms of results, the application of all this tooling allowed us to identify that there is a longitudinal effect in the data. The repetition of news with a high semantic content tends to influence the construction of Clusters, masking relevant information that must be mapped. The analyzes showed that when removing news with the same semantic content, new words emerge, bringing to light a subject that had not been covered until then. Most news stories use the same grouping of words. This grouping occurs due to the repetition of words and the fact that these words are in two or three Clusters. The methodology developed highlights the ability to apply together with the techniques involved in business analysis, competitive data and information, both classic and the most popular contemporary ones, intelligence and competitive strategy.