Micro-entries: Encouraging Deeper Evaluation of Mental Models Over Time for Interactive Data Systems

Jeremy E Block, Eric Ragan

View presentation: 2020-10-25T16:40:00Z GMT-0600 Change your timezone on the schedule page
Exemplar figure, but none was provided by the authors
A graphical representation of capturing and evaluating a user's mental model during system interaction. As users work with an application (1), they are asked to describe noticed patterns and provide their explanations many times (2), which can then be studied by researchers (3). This approach can encourage users to do more reflection of the system performance to reach more informed and less impressionable understandings of system limitations. This capture method provides more comprehensive and higher fidelity accounts of the user's mental model while also tracking how it changes over time.


Many interactive data systems combine visual representations of data with embedded algorithmic support for automation and data exploration. To effectively support transparent and explainable data systems, it is important for researchers and designers to know how users understand the system. We discuss the evaluation of users' mental models of system logic. Mental models are challenging to capture and analyze. While common evaluation methods aim to approximate the user's final mental model after a period of system usage, user understanding continuously evolves as users interact with a system over time. In this paper, we review many common mental model measurement techniques, discuss tradeoffs, and recommend methods for deeper, more meaningful evaluation of mental models when using interactive data analysis and visualization systems. We present guidelines for evaluating mental models over time to allow for assessment of the evolution of specific model updates and mapping to particular use of interface features and data queries. By asking users to describe what they know and how they know it, researchers can collect structured, time-ordered insight into a user's conceptualization process while also helping guide users to their own discoveries.