PNNL

Abstract

Understanding what makes an LLM agent team effective requires systematically evaluating how individual agent configurations influence overall team performance. Multi-Agent Randomized-Block LLM Ensembles (MARBLE) developed a framework for evaluating role-specific model, prompt, and domain-knowledge choices in a technical diagnostic workflow. The project built a reusable agent stack for Generic Pressurized Water Reactor (GPWR) simulator data and a randomized-block tournament method for comparing agent teams under matched conditions. A live FY2026 prompt-team tournament evaluated the two-role Rapid workflow with one fixed candidate model, one Skill preset, and one independent evaluator. It completed 32 candidate executions across three rounds, observed eight of nine possible prompt teams, used about 437,000 tokens, and recorded no execution failures. The leading observed team paired a coverage-focused prompt in the Rapid Diagnostician with an evidence-calibration prompt in the Evidence Adjudicator and reached a mean objective score of 0.924. Twenty-nine of the 32 reports received the evaluator's maximum evidence-quality score, limiting the rubric's ability to distinguish among teams. The study therefore supports a narrow but important conclusion: role-specific prompt assignments influenced observed team performance and token use, although the study did not establish diagnostic accuracy. The project also implemented a blinded, fingerprinted Sequential RBTT lifecycle for future ground-truth evaluation.