VideoPro: A Visual Analytics Approach for Interactive Video Programming

Jianben He, Xingbo Wang, Kam Kwai Wong, Xijie Huang, Changjian Chen, Zixin Chen, Fengjie Wang, Min Zhu, Huamin Qu

DOI: 10.1109/TVCG.2023.3326586

Room: 103

2023-10-24T23:00:00ZGMT-0600Change your timezone on the schedule page
2023-10-24T23:00:00Z
Exemplar figure, described by caption below
The VideoPro interface consists of three major views. The Template View (A) offers descriptive statistics and rich interactions to facilitate multi-faceted exploration and comprehension of labeling templates. The Labeling View (B) provides a summary of the nuanced event compositions within the selected template to allow effective template validation and refinement. It also displays retrieved matching videos for efficient examination and at-scale programming. The Info View (C) presents comprehensive information regarding data embedding distribution in latent space and the model iteration process.
Fast forward
Full Video
Keywords

Interactive machine learning, data programming, video exploration and analysis

Abstract

Constructing supervised machine learning models for real-world video analysis require substantial labeled data, which is costly to acquire due to scarce domain expertise and laborious manual inspection. While data programming shows promise in generating labeled data at scale with user-defined labeling functions, the high dimensional and complex temporal information in videos poses additional challenges for effectively composing and evaluating labeling functions. In this paper, we propose VideoPro, a visual analytics approach to support flexible and scalable video data programming for model steering with reduced human effort. We first extract human-understandable events from videos using computer vision techniques and treat them as atomic components of labeling functions. We further propose a two-stage template mining algorithm that characterizes the sequential patterns of these events to serve as labeling function templates for efficient data labeling. The visual interface of VideoPro facilitates multifaceted exploration, examination, and application of the labeling templates, allowing for effective programming of video data at scale. Moreover, users can monitor the impact of programming on model performance and make informed adjustments during the iterative programming process. We demonstrate the efficiency and effectiveness of our approach with two case studies and expert interviews.