Our groundbreaking mixed reality facial tracking smart glasses application is poised to reshape modern medicine and technology.
- Augmented Reality
- Spatial Design
- User Interface Design
- User Experience Design
The
Background
A private client was looking to develop a facial tracking application with uses in medical and healthcare settings. The goal was to increase the accuracy of consultations and procedures, and provide a new standard of training for students and assistants. They tapped our team at CXR to explore the possibility of a mixed reality smart glasses application – something that had not been done before.
The
Background
The
Challenges
- Mobile headset – This application needed to operate on a mobile processor for obvious reasons – the equipment needed to be readily available and easy to store in an office or classroom setting. Such a complex application had not operated solely on a mobile processor before, so it was unclear if this could be done with the computing power of a portable headset.
- Precise facial mapping – The utility of the application depended on the ability to analyze a face in real time and overlay it with data. For this reason, the tracking needed to use the highest standards of facial tracking to analyze multiple points on the face simultaneously. Hololens 2 architecture does not allow for multi-threaded processing as the Hololens has shifted platforms to ARM64, but several of the key APIs – such as Camera frames – still work on ARM32.
- Understandable interface – Tracking facial features was a challenge in itself, but the interface also needed to be easily understood and make sense for medical students and professionals who may not necessarily be accustomed to Augmented Reality interfaces. Our design team needed to keep this in mind as they created a user experience that relayed complex information in a straightforward design language.
The
Challenges
The
Solution
Engineered with Unity and C++, this revolutionary application can track up to 17 facial feature points at a time, passing camera frames through algorithms and machine learning to overlay a face with an AR user interface in real time.
The
Solution
Features + Process
The application includes a pre-loaded practice face as well, as a cadaver, for training purposes before attempting on an actual patient. This allows the user to become familiar not just with the treatment but the application itself.
The application overlays the data in real time, pinpointing the exact location for procedures. Panels of information are also presented to assist the professional in providing all relevant details to the patient.
While created for injection-based procedures, the tracking and application in general can be reconfigured for operations as important as reconstruction and as common as plastic surgery.
Results
CXR has demonstrated that powerful mixed reality smart glasses applications are now within the realm of possibility for processors such as that of the HoloLens 2, meaning the potential use cases for MR glasses and technology are limitless.
Results
