Viés de gênero na tradução automática: uma avaliação no par inglês-português

Abstract

Neural machine translation (NMT) models have achieved considerable improvements in the quality of their renditions (Castilho; Caseli, 2023). However, such models still present limitations (Caseli, 2017; Castilho; Caseli, 2023), e.g., the reproduction of language that promotes various types of bias (Bianchi et al., 2023), such as gender bias. Gender as a linguistic feature poses a challenge for MT, especially when it comes to translating from a source language that does not realize gender or has notional gender into a language that morphologically realizes grammatical gender (Stanovsky; Smith; Zettlemoyer, 2019; Salvodi et al., 2021). Therefore, evaluating MT systems proves necessary to identify limitations and promote improvements (Popović; Castilho, 2019; Castilho et al., 2023), in addition to mitigating semiotic risks (Matthiessen, 2013) associated to communication mediated by such systems. According to Matthiessen (2013), the notion of semiotic risk encompasses potential problems derived from miscommunication or distortions in communication and in the information flux in semiotic systems, which includes language and other meaning-making media used in information transmission. This thesis reports on a quantitative and qualitative investigation of machine translated sentences. The quantitative approach was conducted by adapting the automatic evaluation code proposed by Stanovsky, Smith and Zettlemoyer (2019) to evaluate translations from the test corpus WinoMT into Portuguese. Concomitantly, the qualitative analysis of the automatic translations was performed from the perspective of Systemic-Functional Theory (Halliday, 1950). To carry out our study , WinoMT, a test corpus designed to evaluate MT models’ capacity of dealing with the grammatical gender of nouns related to professions (Stanovsky; Smith; Zettlemoyer, 2019), was translated from English into Portuguese by the commercial MT models Microsoft Translator (Azure), Amazon Translate (AWS) and Google Translate, as well as the large language model GPT-3.5 Turbo. Results show that the four models perform better in translating occupations (target entities) when the reference gender is male rather than female. In addition, the four models show more accuracy in translating target entities when the reference gender (gold standard) of the occupation reinforces social gender stereotypes. When the target entity is assigned a gender in an anti-stereotypical way, the models generate translations promoting gender bias.