Designing Shared Information Displays for Agents of Varying Strategic Sophistication
Dongping Zhang, Jason Hartline, Jessica Hullman
ACM Computer-Supported Cooperative Work (CSCW) 2024
Diagram of key features of our experimental design. (1) We define a pseudo-Poisson distribution using a Poisson(1.5), which includes L0s-L2s. (2) We endow level-specific beliefs by normalizing the pseudo-Poisson distribution for L1s and L2s who are our study participants. (3) We conduct a staged data collection: in each trial, participants use the information display (shown in Figure 3) to make decisions and then review level-specific feedback (shown in Figure 4). L1s and L2s complete all trials in the same order, but L1s complete the study before L2s, so that L1s’ responses can be used with that of L0s (i.e., the taxi data) to construct level-specific feedback that aligns with L2s’ endowed belief. (4) We calculate the system outcome by combining decisions from L0s (i.e., the taxi data) and L1-L2s (i.e., the collected responses) according to the mixture we used to define the level distribution (step 1). (5) We conduct two replications of the experiment by varying trial order and re-define the level mixture of L0s-L2s using a Poisson(3).
Abstract
Data-driven predictions are often perceived as inaccurate in hindsight due to behavioral responses. We consider the role of interface design choices on how individuals respond to predictions presented on a shared information display in a strategic setting. We introduce a novel staged experimental design to investigate the effects of interface design features, such as the visualization of prediction uncertainty and prediction error, within a repeated congestion game. In this game, participants assume the role of taxi drivers and use a shared information display to decide where to search for their next ride. Our experimental design endows agents with varying level-k depths of thinking, allowing some agents to possess greater sophistication in anticipating the decisions of others using the same information display. Through several large pre-registered experiments, we identify trade-offs between displays that are optimal for individual decisions and those that best serve the collective social welfare of the system. Additionally, we note that the influence of display characteristics varies based on an agent's strategic sophistication. We observe that design choices promoting individual-level decision-making can lead to suboptimal system outcomes, as manifested by a lower realization of potential social welfare. However, this decline in social welfare is offset by a slight reduction in distribution shift, narrowing the gap between predicted and realized system outcomes. This may enhance the perceived reliability and trustworthiness of the information display post hoc. Our findings pave the way for new research questions concerning the design of effective prediction interfaces in strategic environments.