Current methods for evaluation of natural language generation models focus on measuring text quality but fail to probe the model creativity, i.e., its ability to generate novel but coherent text sequences not seen in the training corpus. We present the GenX tool which is designed to enable interactive exploration and explanation of natural language generation outputs with a focus on the detection of memorization. We demonstrate the utility of the tool on two domain-conditioned generation use cases - phishing emails and ACL abstracts.
Published: February 27, 2022
Citation
Duskin K.R., S. Sharma, J. Yun, E.G. Saldanha, and D.L. Arendt. 2021.Evaluating and Explaining Natural Language Generation with GenX. In Workshop on Data Science with Human-in-the-loop: Language Advances (DaSH-LA) colocated with NAACL 2021, June 11, 2021, Virtual, Online, edited by E. Dragut, et al, 70 - 78. Stroudsburg, Pennsylvania:Association for Computational Linguistics.PNNL-SA-159018.