What AI says about CapaBench

CapaBench is an evaluation framework that uses Shapley Value from cooperative game theory to quantify the contributions of individual modules within modular LLM agents, enabling fair attribution and interpretability of their architectures. It evaluates Planning, Reasoning, Action, and Reflection by measuring their marginal and synergistic effects on task performance across a dataset of over 1,500 realistic, multi-round tasks spanning domains like shopping, operation system, robot control, math, and theorem proving. The project intends to open-source the dataset in the future and reports results across multiple models to illustrate how module contributions vary by task and model.