Estudo comparativo entre a capacidade de generalização de modelos criados a partir das principais plataformas de desenvolvimento de assistentes virtuais

Abstract

Chatbots, conversational agents, virtual assistants, or simply bots are used in many cases to represent computer programs with the ability to interact with human beings in a conversational flow, whether by buttons on decision trees or in other more complex cases with the understanding of natural language. Each year, this solution becomes more popular in companies, whether for external audiences such as customers or employees themselves. However, there is a difficulty around organizations when deciding on which platform to use to develop their chatbot. a question arises: is there any difference between the cognitive engines of the main platforms? This work seeks to answer whether there is any difference between the cognitive engines of the three selected chatbots development platforms, Alexa Console Developer from Amazon, Dialogflow from Google, and Watson Assistant from IBM. The factorial experiment compared the ability to generalize Machine Learning models in Brazilian Portuguese for the three platforms, evaluating 5 intentions created in each, divided into two sets, generic and specific. After creating the intentions, the interaction classifications were evaluated by comparing them to the ground truth, and then the difference between performances of each platform, in general, and separated, was verified, adjusting it to a logistic regression model and calculating the Odds Ratio. The experiment showed interesting results, showing that IBM's Watson Assistant has better overall performance, with an average hit percentage of 82% and an Odds Ratio of 4.43 times higher than Dialogflow and 3.03 Alexa Console Developer. However, there was no statistically significant difference in some segregated settings, as in the Class Factor of Intention at the Specific level, where no difference was significant. It is known that the performance factor should not be the only one to be considered and factors such as costs, expertise, ease of implementation, availability, and integration, among others, must also be taken into account. However, as a conclusion of this work, a significant difference was observed between the cognitive engines of the evaluated platforms.