Augmented Reality: Heart Auscultation Application
Background
This project was my first foray into augmented reality (AR) development. As it was my first go, and a school project, I decided to keep my planning loose and play with the technology to see what I could do. I also used it as an opportunity to build my skills in Substance Painter.
Media: Unity Game Engine, 3ds Max, Zbrush, and Substance Painter.
Client: Myself
Audience: UIC Physical Therapy Students
Period: January 2021 - February 2021
Ideation
My initial idea was to use name tags supplied at a hypothetical cardiology conference as image targets to project augmented reality hearts into the chests of conference goers. I knew for sure I wanted a heart so I started by building one that would work on a mobile phone app.
Making the heart
My goal for this sculpt was to create a single model that was high enough fidelity for my demo reel, but also optimized to be used in a mobile phone app. For this project I found some excellent references, especially the Atlas of Cardiac Anatomy provided by the University of Minnesota.
I started with box modeling in 3ds Max. I used Zbrush to clean up the primary form, and then added detail as I increased subdivisions.
One of my goals of this project was to learn Substance Painter, a software I had not previously used. Blending many procedural maps to create detailed materials is exactly my kind of thing. Substance has a new fan!
I used CT data of a human heart as a reference to accurately create the heart beat cycle. I used a morph target modifier with three targets to create the beating animation cycle.
I read that I should aim for about 30k polys for a model used in a phone application. Because it was AR and I expected that could make things a little tougher on the phone’s processing, I created a base mesh of 10k polys. The extra detail is more or less an illusion coming from normal maps, height maps, roughness maps and a carefully painted albedo map.
Augmented reality
Leading up to this project I became familiar with both AR Foundation and Vuforia as a framework for building augmented reality applications. For this project I decided to use Vuforia because I could use a PC webcam as the ‘phone camera’ to live test the app as I worked on it. This was especially useful because it took quite a bit of fine tuning to get the heart to show up in its correct anatomical position.
This was the image target used for the augmented reality app.
Once I started working in Unity I ran into some obstacles with my name tag idea. Different names on different nametags significantly interefered with the app’s ability to identify the target. In addition, name tags can have strong specular highlights. These factors make name tags a non-ideal target for AR image recognition. I found that t-shirt/sweatshirt logos worked much better. They are non-variable and have no sharp specular highlights!
The final app
The proof of concept heart worked great! I just needed to find something to use it for. I played around with a couple of ideas including creating a slider that could change the heart rate. My instructor Sam Bond suggested using the heart as a tool to help physical therapy students learn auscultation sites, which was a great fit for what the app could do. Here is a demo on my roommate!
Bonus
Like I mentioned earlier, I wanted this model to be high enough fidelity to use in my demo reel. Below are some beauty shots rendered with Redshift and composited in After Effects.