Reconhecimento automático de sinais da Libras : desenvolvimento da base de dados MINDS-Libras e modelos de redes convolucionais

Abstract

The automatic recognition of Sign Language has been a challenge for the Computational Intelligence area, given the visual-gestural nature that configures this complex communication system. This thesis falls within this context and focuses efforts on the Brazilian Sign Language, Libras. For this purpose, a new database called MINDS-Libras has been proposed. It contains (i) RGB videos, (ii) videos with depth information, (iii) information from 25 points/joints of the body and from (iv) 1347 points of the face of the signaller. Each of the 20 signs that build this base were recorded 5 times by 12 signallers, totaling 1200 samples. Using this data, two different Deep Learning architectures were proposed for recognizing the MINDS-Libras signs. The first one was a 3D Convolutional Neural Network by using videos, and the second a Temporal Convolutional Neural Network for the manual trajectory. The best leave-one-signaller-out was that based in the hand movement, and this can be considered the most important parameter for sign formation. The results also indicate that this approach is feasible for the Libras signs recognition. New perspectives may be opened with the expansion of the database and add more signallers in the process of recording (new) signs.