Co-Authored Projects

These are projects that I worked on for which I'm not the primary author and was drafted to do a specific part (usually making realtime demos and figures).

The primary author is usually the first person in the list of collaborators (I'll bold them if there IS a primary author at this stage of the project)


Echo-Reconstruction: Audio-Augmented Scene Reconstruction with Mobile Devices (Pending Publication)

Co-investigators: Justin Wilson, Ming Lin, Henry Fuchs

Project summary: Using ML methods and sound reverberation to reconstruct translucent objects like glass based on whether or not audio reflects off of them.

My involvement: Making the video, creating reconstructions, making realtime VR video (the bottom left one is much older)

Text2Gestures: A Transformer Network for Generating Emotive Body Gestures for Virtual Agents

Co-investigators: Uttaran Bhattacharya, Abhishek Banerjee, Pooja Guhan, Aniket Bera, Dinesh Manocha

Project summary: Teach a deep network to create hand and body gestures from text that the avatar is meant to be saying

My involvement: Rig the avatars to work with the generated skeletons, render videos for the user study and final video

I can't yet show the video or paper since it's in review

Tracking and reconstructing full-body process like surgery or physical therapy

Co-investigators: Husam Shaik, Ritika Tejwani, Andrei State, Henry Fuchs

Project summary: reconstruct a person's body prescan and then track and apply their motions over some process to that prescan

My involvement: Advise Shaik and Ritika on the reconstruction/tracking pipeline and make a bunch of tutorials (found on the left).

Realtime hand-based task performance tracking for teaching sign language and surgical knot-tying

Co-investigators: Austin Hale, Manuel Sanchez-casalongue, Timothy Farrell, Henry Fuchs

Project summary: track people's hand during a hand-based dexterity task and provide them realtime feedback and good UX in AR so they can train better

My involvement: Advise Austin on methods for tracking accuracy, skeletal animation, reconstructing things like the rope, user study, etc.

(look at Austin's YT channel found in the left videos for more details of the process)

Redirected Walking with AI-based Deterrents and Distractors for House Tours

Co-investigators: Andrew Fulmer, Surya Poddutoori, Andrei State, Henry Fuchs

Project summary: have virtual avatars in the virtual house that are smart and appropriately move into place to stop the user from leaving the real VR bounds or distract the user to cause extra head rotation.

My involvement: advise Surya and Andrew on RDW methods, navigation, user study, etc.

Ongoing but I'm no longer involved

Eye-tracking for egocentric displays

Co-investigators: Conny Lu, Praneeth Chakravarthula, Henry Fuchs, David Luebke, Andrei State, Agostino Gibaldi, Marty Banks

Project summary: It's an extension of the egocentric reconstruction project that uses eye tracking in an egocentric display like glasses with cameras.

My involvement: Making figures and videos (for now)

Opaque AR displays

Co-investigators: Kishore Rathinavel, Henry Fuchs

Project summary: Making an AR display that can render completely opaque pixels, which also allows us to do shadowing between virtual and physical objects

My involvement: Realtime demo, especially having a physical object cast a shadow on virtual objects.

Customizing walk patterns and locomotion visualizer

Co-investigators: Brennora Cameron, Mary Whitton

Project summary: Getting data and making visualizers that can be used to customize the locomotion method for a particular user, especially for walking-in-place

My involvement: Making the Python visualizer, mentoring/advising, helping with data collection, making the data collection system in Unreal 4 and helping Brennora with the Unity version. This old visualizer used matplotlib, but I have a new one that uses Qt5.

Completed Research Projects

See associated papers under Publications

Optimizing placement of commodity depth cameras for known 3D dynamic scene capture (2015-2016)

Co-investigators: Rohan Chabra, Adrian Illie, Youngwoon Cha, Henry Fuchs

Project summary: Reconstructing surgeries using an array of Kinects around the surgery room

My involvement: Making the synthetic data with Blender, making the realtime implementation and synthetic ground truth with Unity, making the video, being a nurse that gets reconstructed

Effects of virtual acoustics on dynamic auditory distance perception (2017)

Co-investigators: Atul Rungta, Roberta Klatzky, Ming Lin

Project summary: User studies studying how distance perception changes in 3D audio like in VR

My involvement: Helping with user study, making figures in Unreal 4

Supporting free walking in a large virtual environment: imperceptible redirected walking with an immersive distractor (2016-2017)

Co-investigators: Haiwei Chen, Henry Fuchs

Project summary: Getting RDW to work with small tracking areas like in commercial VR using exciting distractors

My involvement: Helping with user study and making video

Glass half full: sound synthesis for fluid–structure coupling using added mass operator (2017)

Co-investigators: Justin Wilson, Auston Sterling, Ming Lin

Project summary: Using physically-based modelling and sound synthesis to allow users to play music by hitting virtual glasses filled with various amount of liquid in specific ways

My involvement: Making the entire realtime implementation in Unreal 4 and the video

Diffraction Kernels for Interactive Sound Propagation in Dynamic Environments (2017-2018)

Co-investigators: Atul Rungta, Carl Schissler, Ravish Mehra, Dinesh Manocha, Oculus

Project summary: Efficiently simulating diffraction using diffraction kernels, which are precomputed files giving information about how propagated sound will distort based on user and sound source positions around a 3D object.

My involvement: Entire realtime implementation in Unreal 4, including 3 demos with predefined paths, a modified version of Oculus' First Contact demo, and a new implementation of Oculus' Toybox demo

Effects of virtual acoustics on target-word identification performance in multi-talker environments (2018)

Co-investigators: Atul Rungta, Carl Schissler, Philip Robinson, Ravish Mehra, Dinesh Manocha, Oculus

Project summary: Seeing if the cocktail party effect exists in VR

My involvement: Making the entire realtime implementation in Unity, made video and some figures

Towards Fully Mobile 3D Face, Body, and Environment Capture Using Only Head-worn Cameras (2018)

Co-investigators: Young-Woon Cha, True Price, Zhen Wei, Xinran Lu, Rohan Chabra, Zihe Qin, Hyounghun Kim, Zhaoqi Su, Yebin Liu, Adrian Ilie, Andrei State, Zhenlin Xu, Jan-Michael Frahm, Henry Fuchs

Project summary: Initial implementation of an egocentric display that gets information about the user (body and face) and the environment so that they can be reconstructed and sent to a different display for telepresence

My involvement: Making the realtime implementation using Unreal 4, Maya, and Alembic. Making video

P-Reverb: Perceptual Characterization of Early and Late Reflections for Auditory Displays (2018-2019)

Co-investigators: Atul Rungta, Roberta Klatzky, Ming Lin

Project summary: Optimizing reverb so that it's suitable for mobile devices

My involvement: Making the figures, video, realtime implementation in Unreal 4, and making multiple tools that compute valid sample points for a scene given certain parameters (see paper)

Audio-Material Reconstruction for Virtualized Reality Using a Probabilistic Damping Model (2019)

Co-investigators: Auston Sterling, Roberta Klatzky, Ming Lin

Project summary: Using data processed from collected material data to allow for a realtime sound synthesis when users hit objects in a 3D scene

My involvement: making video, realtime implementation in Unreal 4

Generating Emotive Gaits for Virtual Agents Using Affect-Based Autoregression (2020)

Co-investigators: Uttaran Bhattacharya, Pooja Guhan, Niall L. Williams, Trisha Mittal, Aniket Bera, Dinesh Manocha

Project summary: Teach a deep network how to create animations describing different emotions, and see if humans can recognize the generated gaits as the correct emotion

My involvement: making the realtime AR implementation, rigging characters, making the 3D parts of the video

Course Projects

CodeQuest (COMP585 Serious Games) (2017)

Co-investigators: Jarrett Grimm, Carmenci Jiminez, Justin Leonard, Diane Brauner

Project summary: Making an accessible iOS game that teaches visually-impaired kids programming. It was extended by a later Serious Games class with my help as LA

My involvement: rewrote pretty much all of the previous team's code and wrote most of the new Swift code for the final implementation

AR Ghost Stories (COMP523 Software Development) (2018)

Co-investigators: Sarah Almond, Gabriel Timotei, Anna Reece, Austin Tinsley, Rachel-Anne Spencer, Brian Moynihan

Project summary: Teaching people about the Dorothea Dix hospital by displaying a ghost that tells its story in the Hololens. The goal was that the user would go through different rooms in the real hospital and different stories would happen. This was extended with my help as LA in a later Serious Games class

My involvement: wrote most of the code and made most of the realtime implementation in Unity. Designed the entire pipeline that gets a reconstruction from ItSeez3D and eventually gets us an animated, lip-synced 3D model.