An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics

Kostiantyn Kucher, Nicole Sultanum, Angel Daza, Vasiliki Simaki, Maria Skeppstedt, Barbara Plank, Jean-Daniel Fekete, Narges Mahyar

View presentation: 2022-10-17T19:30:00Z GMT-0600 Change your timezone on the schedule page
2022-10-17T19:30:00Z
Exemplar figure, described by caption below
Sketch of a visual text analytics design and evaluation workflow with explicit identification of validation concerns which can guide our future efforts for systematic analysis and preparation of guidelines.

The live footage of the talk, including the Q&A, can be viewed on the session page, BELIV: Paper Session 2.

Abstract

Appropriate evaluation and experimental design are fundamental for empirical sciences, particularly in data-driven fields. Due to the successes in computational modeling of languages, for instance, research outcomes are having an increasingly immediate impact on end users. As the gap in adoption by end users decreases, the need increases to ensure that tools and models developed by the research communities and practitioners are reliable, trustworthy, and supportive of the users in their goals. In this position paper, we focus on the issues of evaluating visual text analytic approaches. We take an interdisciplinary perspective from the visualization and natural language processing communities, as we argue that the design and validation of visual text analytics include concerns beyond computational or visual/interactive methods on their own. We identify four key groups of challenges for evaluating visual text analytic approaches (data ambiguity, experimental design, user trust, and big picture concerns) and provide suggestions for research opportunities from an interdisciplinary perspective.