PI: Brenda Praggastis

There seems to be an inverse relationship between the accuracy of deep neural networks (DNNs) and our human ability to understand how they make their decisions and verify their reliability. The Deep Data Profiler (DDP) is a methodology and framework for understanding the fundamental synaptic connections, circuits, and pathways used by trained DNNs for classification tasks.

DDP decomposes a trained DNN into a weighted graph of neurons and synapses and links them to human identifiable concepts, providing both interpretability and trustworthiness for the network. For a given input to a DNN, DDP extracts a profile graph of the most important (activated) neurons in the network and links them by their mutual contributions. By tying an input's profile to the characteristics determining its classification, spurious decisions and poor generalization strategies used by the network can be identified and a measure of trustworthiness can be established.

The idea of linking classification decisions to influential activations for purposes of model interpretability isn't new, but our perspective and approach are novel. DDP analyzes and compares profile graphs using graph theory, topological data analysis, and hypergraph theory to study their structural characteristics and identify a hierarchy of concepts. Once particular concepts are identified as important, their subgraphs of neurons are visualized and tied back to the input domain. We demonstrate the technique through our methodology and have open-source code available to validate our methods and for further research.