Pinpointing Pivot Points: The DECIDE Framework for Data Analysis Decision Importances

Zachary T. Rewolinski, Jan Simson, Frauke Kreuter, Christoph Kern, Bin Yu

Human judgment calls made throughout the data science life cycle can change the conclusions drawn from data. Independent analysts often make different choices when given the same data, leading to often unacceptable variations in results and contributing to the ongoing reproducibility crisis across scientific disciplines. The rapid adoption of agentic tools threatens to amplify this crisis. Several approaches have been developed to address this variation by following the same strategy: repeating an analysis under different combinations of analytical decisions and inspecting the distribution of outcomes. Examples include multiverse analysis, specification curve analysis, and vibration of effects. More broadly, the Predictability-Computability-Stability (PCS) framework assesses how documented judgment calls across the data science life cycle affect conclusions, with predictability serving as a reality-check that a model captures signal. Each of these approaches results in a "multiverse" object. While the distribution of outcomes reflects the overall variability of conclusions, estimating the influence of each individual decision requires additional tools. A common heuristic is to…