MetaExplorer: Facilitating Reasoning with Epistemic Uncertainty in Meta-analysis
Alex Kale, Sarah Lee, Terrance Goan, Elizabeth Tipton, Jessica Hullman
ACM Human Factors in Computing Systems (CHI) 2023
MetaExplorer provides a guided process for literature review and meta-analysis with an emphasis on documenting sources of epistemic uncertainty and choosing how to address them during statistical inference. The workflow proceeds in stages from Scoping and literature review where MetaExplorer elicits information about each study, to Triage and study grouping where the user resolves sources of epistemic uncertainty, and finally to Meta-analysis where the user views results alongside contextualizing uncertainty
Abstract
Scientists often use meta-analysis to characterize the impact of an intervention on some outcome of interest across a body of literature. However, threats to the utility and validity of meta-analytic estimates arise when scientists average over potentially important variations in context like different research designs. Uncertainty about quality and commensurability of evidence casts doubt on results from meta-analysis, yet existing software tools for meta-analysis do not necessarily emphasize addressing these concerns in their workflows. We present MetaExplorer, a prototype system for meta-analysis that we developed using iterative design with meta-analysis experts to provide a guided process for eliciting assessments of uncertainty and reasoning about how to incorporate them during statistical inference. Our qualitative evaluation of MetaExplorer with experienced meta-analysts shows that imposing a structured workflow both elevates the perceived importance of epistemic concerns and presents opportunities for tools to engage users in dialogue around goals and standards for evidence aggregation.