UCLA

Communication barriers between deaf and hearing individuals have led to difficulties in various real-world scenarios, including emergencies, online meetings, and daily interactions. Mobile and Augmented Reality (AR) systems hold promises for offering low-cost solutions for accessible communication. This dissertation aims to bridge the gap between sign language and other oral language users by leveraging assistive Augmented Reality (AR) glasses for everyday wear. We seek to address the full spectrum issues of sign capture, recognition, and language translation on AR glasses and mobile devices.

Our approach first prioritizes the most urgent setting, emergency communication, before extending to more general settings such as daily, online, and learning interactions. We employ domain-specific models derived from sign language with three main components: 1) Capturing sign gestures on a mobile-glass setup using sign parameters; 2) Recognizing signs with lightweight models based on sign parameter correlations and constraints; and 3) Providing general bidirectional ASL-English translation using ASL grammar and word order correlation.

In collaboration with ASL users, our evaluation results demonstrate that, starting from emergency communication scenarios, our domain-oriented models substantially reduce latency by one to two orders of magnitude compared to previous solutions, while maintaining accurate translations. Furthermore, the developed mobile and AR platforms enable sign language interactions across various settings, such as daily in-person communication, virtual communication, and sign language learning, thereby extending the system's applicability. Our research offers an innovative approach to promote accessible communication, fostering social inclusion, and minimizing communication-related inequalities between deaf and hearing individuals.