VR for THA Orthopedic Hip Surgery Simulation

Tools I used

Affiliation

Rowan University VR Center

Team (14)

Dr. Shreekanth Mandayam, Dr. George Lecakes, Prof Amanda Almon, Dr. Tae Won Kim, MD, Meghan Morely, MD, Deep Patel, CMSRU, Alex Wiese, Felicia Veneri, Garrett Williams, Terry Nguyen, Veronica Cava, Lilly Smith, Sofia Monaco, Delano Hendrix

My Role(s):

Lead VR Programmer, Lead Gameplay Programmer, Environment Artist, Prop Artist, Technical Artist

Project Breakdown

One of the unique opportunities of this project was being a team of engineers working with a team of Biomedical Art and Visualization (BMAV) students and faculty. They elevated the visuals of the project by creating realistic hip/leg anatomy in the form of bones, muscles, tendons, etc. Additionally they redesigned the User Interface to be easier to navigate and to be more readable in VR. For both these elements, communication was key between the artists and engineers to get them implemented correctly into the final project. 

For getting the 3D models and textures of the anatomy in the simulation, I created a barebones scene in the Unity project with a similar environment/lighting to the main simulation environment. The artists were given this scene along with a document of instructions on how to import/export their assets in a way that makes them appear accurate to real life. This was a great learning experience for me as I learned how to better communicate the project pipeline. My first iterations of the instructions were partially unclear and resulted in assets being oriented incorrectly or appearing different than the look as seen in Substance Painter. I corrected my mistakes and by the end was happy with both the document and how the assets appeared in the simulation. Thanks to the artists patience their assets were able to be implemented into the simulation with realistic anatomical accuracy.

Additionally for redesigning the UI, I worked with the art team to take their designs from Adobe InDesign and Photoshop to Unity. Thanks to the BMAV team delivering .png files of their icons as well as clearly labelled and organized screenshots of each UI screen, getting the designs implemented was not too difficult. I thought the UI improvements of Unity 4.6 were great, but now the UI tools are robust and very user friendly (go figure). The fonts were easy to implement with Text Mesh Pro, allowing for easy changes to the size, margins, and kerning. By using pure white image files, the icons could have their colors easily set using the Image component without needing to use additional project space for different color variations of the same icon. And finally the swatch library in Unity was a lifesaver when it came to setting the shadows and button colors of each element that each have a specific color on the color pallete. This got us 95% of the way there, but the struggle was the last 5%. Sometimes due to longer strings of information on certain screens meant that the text needed to be altered from the original design, or that the information needed to be condensed. Additionally as new functionality was added to the simulation, the UI designs needed to be modified to accommodate the additional buttons and text fields. At the end of the day I am proud of the team for creating UI/UX designs that made for a better experience in VR.