FlowNL: Asking the Flow Data in Natural Languages

Jieying Huang, Yang Xi, Junnan Hu, Jun Tao

View presentation: 2022-10-21T14:12:00Z GMT-0600 Change your timezone on the schedule page
Exemplar figure, described by caption below
Using FLowNL to explore the crayfish dataset. Users can query flow structure in natural languages using the input box (a). They can resolve unknown terms through conversation in the dialog box (b) based on the existing objects (d). FlowNL system will suggest queries based on templates (e) and display the resulting query formula (c). FlowNL visualizes “tiny spiral flow” in orange, “spiral flow” in green, and “upward flow” in purple (f), respectively, based on user queries.

Prerecorded Talk

The live footage of the talk, including the Q&A, can be viewed on the session page, Natural Language Interaction.

Fast forward

Flow visualization is essentially a tool to answer domain experts' questions about flow fields using rendered images. Static flow visualization approaches require domain experts to raise their questions to visualization experts, who develop specific techniques to extract and visualize the flow structures of interest. Interactive visualization approaches allow domain experts to ask the system directly through the visual analytic interface, which provides flexibility to support various tasks. However, in practice, the visual analytic interface may require extra learning effort, which often discourages domain experts and limits its usage in real-world scenarios. In this paper, we propose FlowNL, a novel interactive system with a natural language interface. FlowNL allows users to manipulate the flow visualization system using plain English, which greatly reduces the learning effort. We develop a natural language parser to interpret user intention and translate textual input into a declarative language. We design the declarative language as an intermediate layer between the natural language and the programming language specifically for flow visualization. The declarative language provides selection and composition rules to derive relatively complicated flow structures from primitive objects that encode various kinds of information about scalar fields, flow patterns, regions of interest, connectivities, etc. We demonstrate the effectiveness of FlowNL using multiple usage scenarios and an empirical evaluation.