TactualPlot: Spatializing Data as Sound using Sensory Substitution for Touchscreen Accessibility

Pramod Chundury, Yasmin Reyazuddin, J. Bern Jordan, Jonathan Lazar, Niklas Elmqvist

Room: 105

2023-10-25T23:00:00ZGMT-0600Change your timezone on the schedule page
Exemplar figure, described by caption below
Teaser with 3 composite images. Image A shows a mockup of the tactual plot technique. Image B shows a screenshot of the TactualPlot system implemented on an iPad. A square-shaped scatterplot is shown at the bottom of the screen, and contains the data points, and the axes. The top part of the screenshot displays buttons that control the prototype. Image C shows the same scatter plot that has been printed on thermoform paper, and is overlaid on the iPad.
Fast forward
Full Video

Accessibility, sonification, multimodal interaction, crossmodal interaction, visualization


Tactile graphics are one of the best ways for a blind person to perceive a chart using touch, but their fabrication is often costly, time-consuming, and does not lend itself to dynamic exploration. Refreshable haptic displays tend to be expensive and thus unavailable to most blind individuals. We propose TactualPlot, an approach to sensory substitution where touch interaction yields auditory (sonified) feedback. The technique relies on embodied cognition for spatial awareness—i.e., individuals can perceive 2D touch locations of their fingers with reference to other 2D locations such as the relative locations of other fingers or chart characteristics that are visualized on touchscreens. Combining touch and sound in this way yields a scalable data exploration method for scatterplots where the data density under the user’s fingertips is sampled. The sample regions can optionally be scaled based on how quickly the user moves their hand. Our development of TactualPlot was informed by formative design sessions with a blind collaborator, whose practice while using tactile scatterplots caused us to expand the technique for multiple fingers. We present results from an evaluation comparing our TactualPlot interaction technique to tactile graphics printed on swell touch paper.