Mega AI

PI: Maria Glenski

Objective

Scaling artificial intelligence (AI) to massive-scale, multi-purpose foundation models capable of multi-purpose inference to enable reasoning and generative tasks for science and security mission domains.

  • Climate fiscal year (FY) 2022 and chemistry (FY22, FY23)
    • pretraining large-scale AI models from scratch (1B+ params)
    • scientific knowledge: text (0.87TB), molecular databases (110K, properties + structures)
  • Cybersecurity and code (FY24)
    • targeted fine-tuning and adaption to boost performance on multi-purpose tasking from zero-shot to instruction tuned
    • enable on-prem, mission-informed vulnerability assessment and code characterization, search, and assessment
An overview of several task types supported by our Chemistry model, with an example for chemical entity extraction where domain-pretrained models can outperform larger, open-source SOTA models.
An overview of several task types supported by our Chemistry model, with an example for chemical entity extraction where domain-pretrained models can outperform larger, open-source models.

Overview 

State-of-the-art, large-scale language models or multimodal foundation models incorporating a text modality are trained on large collections of pretraining data, largely focusing on general-purpose language and/or vision data sources. However, performance often degrades when applying foundation models trained on general-purpose datasets to science and security domains, such as handling the vocabulary shift between general language versus domain knowledge in areas like molecular chemistry and climate. By leveraging a large collection of scientific literature, Mega AI focuses on developing next-generation foundation models addressing research gaps in large-scale multimodal representation learning, multitask inferences, and generalizability, rapid adaptivity, and usability of these artificial intelligence technologies.

Foundation models, also known as neural platforms, are large-scale artificial intelligence models that leverage self-supervised pretraining at scale—for example, large-scale pretraining with zero supervision on unlabeled data—and can support adaptation to a wide range of downstream tasks via transfer learning and fine-tuning toward specific tasks.

Analyses performed on the Mega AI project have measured the benefit of in-domain pretraining when leveraging foundation models, including highlighting the performance gains over both general-language-focused baseline models and general-language baselines, which received additional pretraining on the Mega AI pretraining collection.

Scaling AI for science and security

Impact

Other conventional open-source or commercial approaches may require:

  • application programming interface (API) usage (cloud-based querying)
  • restrictions on adaption, tuning, or appropriate use
  • lack of transparency on (proprietary) data or training choices that could impact usability or quality of models if used for mission

Mega AI explores tradeoffs of development choices (pretraining from scratch, fine-tuning off-the-shelf base models, and targeted fine-tuning and/or task-prompts) and model performance to support on premise model use, mission informed training/tuning of usable large language models (LLMs), and traceable model development and evaluation.

Publications and Presentations

  • Horawalavithana Y.S., E.M. Ayton, S. Sharma, S.A. Howland, M. Subramanian, S.W. Vasquez, and R.J. Cosbey, et al. 2022. "Foundation Models of Scientific Knowledge for Chemistry: Opportunities, Challenges and Lessons Learned." In Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models, May 2022, Vitrual and Dublin, Ireland, 160–172. Stroudsburg, Pennsylvania: Association for Computational Linguistics. doi:10.18653/v1/2022.bigscience-1.12
  • COLING 2022 Workshop on Multimodal/Multipurpose AI Evaluation, Artificial Intelligence for Earth System Predictability (AI4ESP). Volkova S., N.O. Hodas, and T.D. Scheibe. 2021. "AI-Driven Cross-Domain Knowledge Discovery and Hypotheses Generation for Enhanced Earth System Predictability."