Pure Math for Machine Learning

Objective 

  • We leverage kernel machines as explainable surrogates for modern learners such as neural networks.
  • Every neural network has natural kernel surrogates
    • Neural Tangent Kernels are linear approximations to the whole network
    • Conjugate Kernels leverage the embedded features of the final activations
  • Ultimately, we can use linear model theory on these surrogates to interpret results of the machine learning model.
Identification of batch effects in data: We can use the feature representation to distinguish between human curated (red) vs machine (blue) curated sentiment in text. 
Identification of batch effects in data: We can use the feature representation to distinguish between human curated (red) vs machine (blue) curated sentiment in text. 

Overview 

PMML

The Pure Mathematics in Machine Learning (PMML) team provides basic research on the explainability and interpretability of neural networks. Approximately half the project focuses on mathematical theory, while the other half performs experiments to obtain heuristic explanations for training neural networks. The main tool used in our research is kernel approximations for neural networks, specifically the neural tangent and conjugate kernels. In our theoretical studies, we train small feedforward networks and prove theories about the spectral distribution of these kernels that add understanding to the global convergence of the models and contrast the efficacy between these two kernels.

For our empirical studies, we extracted kernels from both shallow and deep networks and trained kernel generalized linear models (GLMs) and the support vector machine algorithm to show that these kernelized GLMs are almost an identical pointwise approximation for classification networks. This proxy allows us to use existing statistical tools to understand the internal learning mechanisms of the neural network.

Impact

We have shown that linear approximations are good enough for actionable insight, which:

  • provide an explanation of decisions via first order approximation 
  • allow the analyst to use all the traditional data science tools they understand for exploration (principal component analysis [PCA], regression, classification, clustering)

Publications and Presentations

  • Howard A.A., S. Qadeer, A.W. Engel, A.Y. Tsou, M. Vargas, T.Y. Chiang, and P. Stinis. 2024. "The conjugate kernel for efficient training of physics-informed deep operator networks." In ICLR 2024 Workshop on AI4DifferentialEquations In Science.
  • A.W. Engel, Z. Wang, N. Frank, I. Dumitriu, S. Choudhury, A.D. Sarwate, and T.Y. Chiang. 05/07/2024. "Faithful Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models." International Conference on Learning Representations 2024 Main Track Spotlight, Vienna, Austria.
  • Chiang T.Y., A. Harlev, A.W. Engel, and P. Stinis. 12/10/2023. "Exploring Learned Representations of Neural Networks with Principal Component Analysis." New Orleans, Louisiana.