Assistive AR for Dinner Table Conversations
A Magic Leap One prototype exploring how spatial captions could help hard-of-hearing family members follow and participate in group conversations.
This project explored how augmented reality could help hard-of-hearing people feel more included in group conversations.
The idea came from a real family story: someone at a dinner table could not hear enough of the conversation to follow along, respond naturally, or feel fully present. We used that scenario as the starting point for an assistive AR concept using the newly released Magic Leap One headset.
The concept was simple: show speech as spatial captions or speech bubbles near the people speaking, so a hard-of-hearing user could visually follow the conversation and participate more easily.
I pitched the original concept at a VR hackathon, inspired by a friend's story about family dinner conversations and hearing loss.
I worked as a UX designer and Unity technologist on the project. My contributions included:
- Framing the assistive technology concept
- Researching hearing loss, social connection, and dinner table conversation challenges
- Brainstorming scenarios with the team
- Sketching and whiteboarding the experience
- Liaising with Magic Leap employees to understand the device's capabilities and limits
- Designing 3D speech bubble objects in Fusion 360
- Supporting Unity implementation and demo setup
- Helping shift the project into a Wizard of Oz prototype when real-time speech-to-text became impractical
- Participating in the live demo and presentation
The challenge was to create a meaningful assistive technology demo using brand new hardware with real technical constraints.
We wanted to show the potential of Magic Leap One as more than a novelty device. The concept needed to connect to a real-world problem, make use of spatial AR, and feel understandable to people trying the demo.
The technical challenge was that the headset and software were still new. We had limited access to the device at first, and some of our early assumptions did not hold up. In particular, the internal microphone was better suited to capturing the wearer's voice than someone across the table in a noisy room. That made our original real-time speech-to-text plan difficult to implement during the hackathon.
We began by brainstorming assistive technology ideas based on stories from family and friends. The dinner table scenario became the strongest direction because it was specific, emotional, and easy to understand. The problem was not only hearing the words. It was also about belonging. If a person misses too much of a shared conversation, they can become isolated even while sitting with family.
We looked into the prevalence and effects of hearing loss, including how it can affect connection, confidence, and participation. That research helped us frame the prototype around inclusion rather than only transcription. The goal became helping someone re-enter a conversation, not just showing text.
Because the Magic Leap One had just been released, part of the process was learning what the device could do and what had been implemented at this early date. We explored the headset's spatial mapping, room scanning, object placement, display behavior, and Unity workflow. We also spoke with Magic Leap employees at the event to better understand what was possible within the time and technical limits.
We sketched and whiteboarded different scenarios for how spatial captions might appear. We focused on the family dinner table because it gave us a clear scene, clear roles, and a specific emotional goal. Role-playing helped us understand where captions should appear, how the user might turn between speakers, and how the experience could help someone participate instead of only observe.
I designed several 3D speech bubble forms in Fusion 360 to use as backgrounds for captions. The idea was to support different lengths of spoken text with different bubble sizes. The visual treatment needed to be readable in the headset, spatially stable, and easy to understand as part of a conversation.
Our early plan was to use real-time speech-to-text, but the audio capture and device constraints made that unrealistic for the hackathon. Instead, we created a Wizard of Oz demo. We wrote a short dinner table script and set up stationary virtual speech bubbles that could be shown and hidden in sequence. A person behind the scenes advanced the captions after each spoken line. Magic Leap's room scanning helped us lock the speech bubbles in space above the table, which made the demo feel tied to the real dinner table environment.
The scenario made the problem immediately understandable. People could imagine the frustration of missing a family conversation.
Instead of placing text only at the bottom of the display, we explored captions that existed in the room near the conversation. This helped connect the words to the social context.
The Wizard of Oz method let us test and communicate the core idea without needing fully functional real-time speech recognition.
The demo was not only about watching captions appear. The participant was asked a question so they could experience rejoining the conversation.
The key moment was not technical. It was the moment people realized the system could show them what was being said and help them feel included.
The final demo placed participants in the role of someone who was hard of hearing at a family dinner table. Participants wore earplugs while two team members spoke quietly nearby. They could not clearly hear the conversation, but when they put on the Magic Leap headset, spatial speech bubbles appeared above the table in sync with the scripted conversation.
The project received the Award for Outstanding Novelty. Judges and attendees responded strongly to the idea, often connecting it to their own family experiences. Several people immediately understood the promise of using AR to support communication, inclusion, and accessibility.
One of the most memorable reactions came when people realized the system could show them what was being said. That moment showed that the concept was landing clearly.
This project reinforced that emerging technology is most compelling when it starts with a real human problem.
It also showed the value of Wizard of Oz prototyping. Even though the speech-to-text system was not fully functional, the prototype let people experience the intended interaction, emotional context, and assistive potential.
The next step would be to test real speech-to-text options, compare latency and accuracy, and understand how captions should behave in real conversations with multiple speakers. A future version would also need to solve speaker identification and wearability.