Reconhecimento ativo de pequenos objetos pela fusão de dados audiovisuais

Abstract

Robots routinely face the need to recognize common use objects, be it for domestic use, search and rescue tasks or surveillance systems. This ability fundamentally requires them to process sensory information and best represent it, in order to maximize its performance. This work presents an active perception approach to object recognition using both audio and visual stimuli, acquired by sensors mounted on a robot, which uses an articulated rod to poke the object in order to actively generate audio signatures. The object domain consists of a structured set of small objects, in which simple geometries and single-material compositions are adopted in order to make it easier to achieve a comprehension of the make-up of audio signatures. For each combination of geometry and material composition, an audiovisual signature is developed in a machine learning approach that implements sensor fusion. Performance of classification is evaluated for the original signals and for decreasing signal-to-noise ratio of the audio signals, where two strategies for sensor fusion are comparatively evaluated: decision fusion in a meta-learning manner, and feature fusion. Decision fusion is shown to perform best and improves over audio- or video-only classification, with accuracies of 98.6%, 96.2%, and 95.1%, respectively, enhancing recognition and providing stability over high interference scenarios. The audio descriptors introduced are ranked according to their discriminatory power. Contributions of this work includes evaluation of techniques for representation of impulsive signals, a framework for audiovisual fusion and the release of the dataset.