RL-LABEL: A Deep Reinforcement Learning Approach Intended for AR Label Placement in Dynamic Scenarios



Zhutian Chen, Daniele Chiappalupi, Tica Lin, Yalong Yang, Johanna Beyer, Hanspeter Pfister

 DOI: 10.1109/TVCG.2023.3326568

 Room: 109

2023-10-26T05:33:00ZGMT-0600Change your timezone on the schedule page
2023-10-26T05:33:00Z

Exemplar figure, described by caption below — RL-LABEL not only adapts label placement considering players’ current motion status (e.g., speed, direction) and long-term outcomes but also ensures label stability over time. (a) Both players move left, with the rear player moving faster. A label is attached to the front player. (b)-(c) Using a force-based method, the label shifts left to avoid immediate occlusion but results in future occlusion. (d)-(e) With our method, the label moves right, sacrificing some immediate occlusion-free space for preventing future occlusion and ensuring stable visibility.

Fast forward

Keywords

Augmented Reality, Reinforcement Learning, Label Placement, Dynamic Scenarios

Abstract

Labels are widely used in augmented reality (AR) to display digital information. Ensuring the readability of AR labels requires placing them occlusion-free manner while keeping visual linkings legible, especially when multiple labels exist in the scene. Although existing optimization-based methods, such as force-based methods, are effective in managing AR labels in static scenarios, they often struggle in dynamic scenarios with constantly moving objects. This is due to their focus on generating layouts optimal for the current moment, neglecting future moments and leading to sub-optimal or unstable layouts over time. In this work, we present RL-LABEL, a deep reinforcement learning-based method intended for managing the placement of AR labels in scenarios involving moving objects. RL-LABEL considers both the current and predicted future states of objects and labels, such as positions and velocities, as well as the user’s viewpoint, to make informed decisions about label placement. It balances the trade-offs between immediate and long-term objectives. We tested RL-LABEL in simulated AR scenarios on two real-world datasets, showing that it effectively learns the decision-making process for long-term optimization, outperforming two baselines (i.e., no view management and a force-based method) by minimizing label occlusions, line intersections, and label movement distance. Additionally, a user study involving 18 participants indicates that, within our simulated environment, RL-LABEL excels over the baselines in aiding users to identify, compare, and summarize data on labels in dynamic scenes.