Old Wine in a New Bottle? Analysis of Visual Lineups with Signal Detection Theory
Sheng Long, Matthew Kay
BELIV 2024
The image connects a visual lineup task with signal detection theory. It shows a lineup of multivariate images where participants identify if one differs or if there's "no discernible difference." Signal detection theory analyzes this data, assuming perceptual evidence for signal presence/absence as overlapping probability distributions. This quantifies observer sensitivity and decision criterion, separating perceptual sensitivity from response bias. The graphs illustrate concepts like false alarm rate, hit rate, and sensitivity (d'), demonstrating how the theory applies to perceptual decision-making in visual discrimination tasks.
Abstract
This position paper critically examines the graphical inference framework for evaluating visualizations using the lineup task. We present a re-analysis of lineup task data using signal detection theory, applying four Bayesian non-linear models to investigate whether color ramps with more color name variation increase false discoveries. Our study utilizes data from Reda and Szafir’s previous work [20], corroborating their findings while providing additional insights into sensitivity and bias differences across colormaps and individuals. We suggest improvements to lineup study designs and explore the connections between graphical inference, signal detection theory, and statistical decision theory. Our work contributes a more perceptually grounded approach for assessing visualization effectiveness and offers a path forward for better aligning graphical inference methods with human cognition. The results have implications for the development and evaluation of visualizations, particularly for exploratory data analysis scenarios. Supplementary materials are available at https://osf.io/xd5cj/.