PI: Ben Wilson

Nuclear non-proliferation analysis is complex and subjective. The data in this domain is sparse and examples are both rare and diverse. Hypotheses and recommendations extracted from text reporting need to be linguistic in nature so that there is enough detail for analysts. Further, the finding must also be completely auditable so any claim or assertion can be sourced directly to the reference material from which it was derived. This painstaking process is currently accomplished by analysts thoroughly documenting underlying assumptions and clearly referencing details to source documents. This manual process is labor-intensive and does not scale well with rapidly increasing quantities of data.

Recent breakthroughs in natural language processing techniques that use sophisticated model architectures and extensive pretraining have been able to demonstrate state-of-the-art performance on a variety of language-based tasks across large, heterogeneous data sets. Furthermore, these breakthroughs have led to language models that not only model language syntax but contain relational knowledge present in the training data, greatly enhancing our ability to perform question and answering tasks. Though they are trained in a self-supervised manner on language-modeling tasks, these models have proven to surpass specifically trained question-answering systems on benchmark data sets.

Researchers at PNNL are developing a method to audit the answers provided by these language models against the texts in the underlying training data. This capability, combined with the existing question answering capabilities of large language models, enables researchers to create models that greatly assist nuclear non-proliferation analysts. Furthermore, they combine this new audit function with a language model, subject matter expert query set, and visualization interface that is customized for use by nuclear non-proliferation analysts.