EU Horizon 2020
Horizon 2020
HomeNewsResearch ThemesPeopleKey Prior PublicationsPublications
[LK22] Emanuele La Malfa and Marta Kwiatkowska. The King is Naked: on the Notion of Robustness for Natural Language Processing. In Proc. 36th AAAI Conference on Artificial Intelligence (AAAI'22). To appear. March 2022. [pdf] [bib]
Downloads:  pdf pdf (1.23 MB)  bib bib
Abstract. There is growing evidence that the classical notion of adversarial robustness originally introduced for images has been adopted as a 𝑑𝑒 π‘“π‘Žπ‘π‘‘π‘œ standard by a large part of the NLP research community. We show that this notion is problematic in the context of NLP as it considers a narrow spectrum of linguistic phenomena. In this paper, we argue for π‘ π‘’π‘šπ‘Žπ‘›π‘‘π‘–π‘ π‘Ÿπ‘œπ‘π‘’π‘ π‘‘π‘›π‘’π‘ π‘ , which is better aligned with the human concept of linguistic fidelity. We characterize π‘ π‘’π‘šπ‘Žπ‘›π‘‘π‘–π‘ π‘Ÿπ‘œπ‘π‘’π‘ π‘‘π‘›π‘’π‘ π‘  in terms of biases that it is expected to induce in a model. We study π‘ π‘’π‘šπ‘Žπ‘›π‘‘π‘–π‘ π‘Ÿπ‘œπ‘π‘’π‘ π‘‘π‘›π‘’π‘ π‘  of a range of π‘£π‘Žπ‘›π‘–π‘™π‘™π‘Ž and robustly trained architectures using a template-based generative test bed. We complement the analysis with empirical evidence that, despite being harder to implement, π‘ π‘’π‘šπ‘Žπ‘›π‘‘π‘–π‘ π‘Ÿπ‘œπ‘π‘’π‘ π‘‘π‘›π‘’π‘ π‘  can improve performance on complex linguistic phenomena where models robust in the classical sense fail.