Facilitating Conversational Interaction in Natural Language Interfaces for Visualization

Rishab Mitra, Arpit Narechania, Alex Endert, John Stasko

View presentation:2022-10-19T19:00:00ZGMT-0600Change your timezone on the schedule page
2022-10-19T19:00:00Z
Exemplar figure, described by caption below
Natural language (NL) toolkits enable visualization developers, who may not have a background in natural language processing (NLP), to create natural language interfaces (NLIs) for end-users to flexibly specify and interact with visualizations. NL4DV is one such Python-based toolkit that processes an NL query about a tabular dataset and returns an analytic specification containing data attributes, analytic tasks, and relevant visualizations, modeled as a JSON object. We extend NL4DV to enable developers to integrate powerful conversational interaction capabilities, e.g., facilitate a multi-turn dialog between the user and the system instead of one-off utterances. For example, given a dataset on houses, a user asks, “Show mean prices for different home types over the years.” In response, NL4DV recommends a multi-series line-chart visualization. Desiring a bar chart instead, the user asks a follow-up query, “As a bar chart” and NL4DV outputs a bar chart visualization. Finally, desiring only certain home types, the user asks, “Just show condos and duplexes.” NL4DV outputs a filtered visualization.

Prerecorded Talk

The live footage of the talk, including the Q&A, can be viewed on the session page, Visualization Systems and Graph Visualization.

Keywords

Natural Language Interfaces; Visualization Toolkits; Conversational Interaction

Abstract

Natural language (NL) toolkits enable visualization developers, who may not have a background in natural language processing (NLP), to create natural language interfaces (NLIs) for end-users to flexibly specify and interact with visualizations. However, these toolkits currently only support one-off utterances, with minimal capability to facilitate a multi-turn dialog between the user and the system. Developing NLIs with such conversational interaction capabilities remains a challenging task, requiring implementations of low-level NLP techniques to process a new query as an intent to follow-up on an older query. We extend an existing Python-based toolkit, NL4DV, that processes an NL query about a tabular dataset and returns an analytic specification containing data attributes, analytic tasks, and relevant visualizations, modeled as a JSON object. Specifically, NL4DV now enables developers to facilitate multiple simultaneous conversations about a dataset and resolve associated ambiguities, augmenting new conversational information into the output JSON object. We demonstrate these capabilities through three examples: (1) an NLI to learn aspects of the Vega-Lite grammar, (2) a mind mapping application to create free-flowing conversations, and (3) a chatbot to answer questions and resolve ambiguities.