Milliways: Taming Multiverses through Principled Evaluation of Data Analysis Paths
Abhraneel Sarma, Kyle Hwang, Jessica Hullman and Matthew Kay
ACM Human Factors in Computing Systems (CHI) 2024
The Milliways interface consists of four panels to support the multiverse analysis tasks and a principled evaluation of the multiverse analysis. The outcome panel (A1) presents the results of the multiverse—for each universe in the multiverse, we show the outcome. The specification panel (A2) reveals the options for each parameter which the universe is composed of. The code panel (A3) presents the code that was used to construct the multiverse (here, the code was from the multiverse R library). The data panel (A4) shows the dataset used in the analysis. Other elements include the add button to add additional outcome variables (A5) , exclude button to remove options of a parameter (A6) , link button to link (aggregate) the outcomes across two or more options (A7) , slider for hierarchical sorting based on average effect (A8) and the zoom toggle (A9).
Abstract
Multiverse analyses involve conducting all combinations of reasonable choices in a data analysis process. A reader of a study containing a multiverse analysis might question—are all the choices included in the multiverse reasonable and equally justifiable? How much do results vary if we make different choices in the analysis process? In this work, we identify principles for validating the composition of, and interpreting the uncertainty in, the results of a multiverse analysis. We present Milliways, a novel interactive visualisation system to support principled evaluation of multiverse analyses. Milliways provides interlinked panels presenting result distributions, individual analysis composition, multiverse code specification, and data summaries. Milliways supports interactions to sort, filter and aggregate results based on the analysis specification to identify decisions in the analysis process to which the results are sensitive. To represent the two qualitatively different types of uncertainty that arise in multiverse analyses—probabilistic uncertainty from estimating unknown quantities of interest such as regression coefficients, and possibilistic uncertainty from choices in the data analysis—Milliways uses consonance curves and probability boxes. Through an evaluative study with five users familiar with multiverse analysis, we demonstrate how Milliways can support multiverse analysis tasks, including a principled assessment of the results of a multiverse analysis.