XR Multi-Application Management

For my Master's capstone in Human Centered Design & Engineering at the University of Washington, I worked with Camila Proffitt and Siddhant Patil on a research and design project sponsored by Pluto VR.

We began with a broad question: what interaction problems were emerging as spatial computing environments became more complex?

Early interviews with Pluto VR helped us narrow the project toward multi-application management. Their work involved communication and collaboration across AR and VR platforms, and they were already encountering challenges around managing multiple tools, objects, applications, and shared resources in three-dimensional space.

That led us to focus on how people might find, open, hide, minimize, quit, and switch between applications in AR, VR, XR, and future blended environments.

On desktop systems, people rely on familiar tools such as Finder, Windows Explorer, docks, windows, menus, folders, and task switchers. In spatial computing, those models do not translate directly. Applications may exist behind the user, outside their field of view, obscured by other objects, placed elsewhere in a virtual room, or layered over the physical world.

Our goal was to identify the core tasks a spatial application-management system would need to support, then explore interaction models that could work across platforms rather than for one specific headset, operating system, or input method.

I contributed throughout the project, from initial research and problem definition through concept development, prototyping, expert evaluation, and public documentation. My main contributions included:

• Participating in interviews with Pluto VR and other spatial computing experts
• Researching interaction patterns across AR and VR systems
• Comparing how existing platforms handled visibility, navigation, placement, and control
• Sketching and refining interaction concepts with the team
• Leading creation of the VR video prototype
• Building scenes and interface concepts in Microsoft Maquette and Unity
• Recording first-person footage from inside a VR headset
• Presenting concepts through sketches, images, video, and live headset screen sharing
• Contributing to the Medium article, YouTube video, and capstone documentation

We began with onsite interviews at Pluto VR.

Rather than starting with a predefined interface solution, we asked about the problems their team was seeing in spatial computing and the challenges they expected to become more important as people worked across multiple applications.

Those conversations helped us identify multi-application management as a useful area of focus.

The problem was broader than creating an application launcher. Users might need to locate an application outside their field of view, understand which applications were active, hide or minimize something without losing it, return to an application later, and switch between tasks without becoming disoriented.

This moved the project from a broad investigation of spatial-computing problems to a focused exploration of multi-application workflows.

Once we had defined the problem area, we studied how existing AR and VR systems approached related tasks.

Using devices available through the University of Washington lab, Pluto VR, and my own equipment, we reviewed applications and interaction models across different headsets.

We collected screenshots, notes, and examples related to:

• Application launchers
• Menus and navigation
• Object placement
• Visibility and occlusion
• Selection methods
• Spatial orientation
• Minimizing and restoring content
• Hand, controller, gaze, and pointer input
• Visual feedback and application state

We also reviewed foundational HCI principles and considered how ideas developed for desktop and mobile systems might change in spatial environments.

This research helped us distinguish between familiar desktop concepts that remained useful and patterns that needed to be rethought for three-dimensional space.

Each team member sketched possible interaction models independently, then we reviewed and refined them together.

We explored ways to help users:

• Find applications around them
• View active and available applications
• Minimize and restore content
• Switch between tasks
• Understand application state
• Organize applications spatially
• Reduce unnecessary movement
• Maintain orientation as content appeared, moved, or disappeared

Because spatial interaction is physical, we often acted out the gestures and movements we were proposing.

We pointed, turned, reached, opened imaginary menus, and mimed repeated interactions to understand how they might feel over time. This helped us identify concepts that looked reasonable in a sketch but felt awkward, tiring, unclear, or physically excessive when performed repeatedly.

The concepts were treated as hypotheses based on the research, not as finished solutions.

Our original plan included in-person usability testing with participants using AR and VR headsets.

The project took place during the beginning of COVID, when bringing people into a lab and sharing physical headsets was no longer practical.

Remote testing with people who had little or no headset experience introduced another problem. Much of the session could have been spent learning the device rather than evaluating the multi-application interaction concepts.

We chose to work with experienced spatial computing developers and subject matter experts instead, including people from Pluto VR, Microsoft, and the broader XR field.

Their familiarity with the platforms allowed them to focus on whether the workflows addressed meaningful problems, whether the interaction models were understandable, and whether the concepts appeared technically and behaviorally plausible.

This was expert evaluation rather than broad usability validation.

We showed the work in increasingly detailed forms as the project developed.

We first shared sketches and early design directions. This helped us discuss the problem and compare alternatives before investing heavily in one solution.

As the concepts became clearer, we created interface images showing layouts, menus, application states, and interaction sequences.

I then created spatial scenes and interface concepts using Microsoft Maquette and Unity. I recorded first-person footage from inside a headset and added voiceover to explain the proposed interactions.

I also shared my live headset view during remote sessions so reviewers could see how the concepts appeared from the user's perspective and discuss the spatial experience in real time.

Using sketches, interface images, video, and live screen sharing helped us move from broad conceptual feedback toward more focused evaluation of the interaction models.

The feedback helped us assess:

• Whether the proposed tasks reflected real platform problems
• Whether application states were clear
• Whether users could maintain orientation
• Whether interactions required too much movement
• Where desktop metaphors remained useful
• Where additional visual or behavioral feedback was needed
• Which concepts depended too heavily on one platform

The feedback informed how we refined and presented the design directions, while also clarifying what would still require broader testing.

The project resulted in a set of research findings, design principles, interaction concepts, prototype materials, and public-facing documentation.

We published a Medium article describing six considerations for XR multi-application management and a YouTube video presenting the research, process, and prototype concepts.

The project demonstrated a complete exploratory research and design process:

• Sponsor interviews
• Problem definition
• Comparative and foundational research
• Interaction design
• Progressive prototyping
• Expert evaluation
• Concept refinement
• Public documentation

Medium

Read on Medium

The article presents the main research findings and design considerations from the project.