Imagining Replications: Graphical Prediction & Discrete Visualizations Improve Recall & Estimation of Effect Uncertainty
Jessica Hullman, Matthew Kay, Yea-Seul Kim, Samana Shrestha
IEEE Trans. Visualization & Comp. Graphics (Proc. InfoVis), 2017
A depiction of distributions relevant to replication uncertainty, including those based on perfect knowledge of the world (left) and those derived from samples obtained in experimentation (right). Our study evaluated whether graphically predicting the true distribution one would see if an experiment were replicated many times using either discrete or continuous visualizations impacted how well participants could recall an observed sampling distribution they were shown (6) or estimate the uncertainty in the replication prediction distribution (7) for a new study.
Abstract
People often have erroneous intuitions about the results of uncertain processes, such as scientific experiments. Many uncertainty visualizations assume considerable statistical knowledge, but have been shown to prompt erroneous conclusions even when users possess this knowledge. Active learning approaches have been shown to improve statistical reasoning, but are rarely applied in visualizing uncertainty in scientific reports. We present a controlled study to evaluate the impact of an interactive, graphical uncertainty prediction technique for communicating uncertainty in experiment results. Using our technique, users sketch their prediction of the uncertainty in experimental effects prior to viewing the true sampling distribution from an experiment. We find that having a user graphically predict the possible effects from experiment replications is an effective way to improve one’s ability to make predictions about replications of new experiments. Additionally, visualizing uncertainty as a set of discrete outcomes, as opposed to a continuous probability distribution, can improve recall of a sampling distribution from a single experiment. Our work has implications for various applications where it is important to elicit peoples’ estimates of probability distributions and to communicate uncertainty effectively.