Immersive Stories Lab

The Immersive Stories Lab is a unique space among liberal arts colleges that is open to students, faculty, and staff who wish to tell their stories using immersive projection, motion capture, virtual reality, or augmented reality. Computer Science faculty member William Bares started this lab in Fall 2020 and welcomes collaborations. As of summer 2023, we have welcomed collaborations from Art, Classics, Dance, Film and Media Studies, Geology, Hispanic Studies, The Sheehan Art Gallery, and Theater. The lab is located in Olin Hall behind the Computer Science commons space. Situating such a lab in a small liberal arts college where it takes just a couple of minutes to walk from one building to the next encourages collaborations. Additionally, Olin Hall is shared by Computer Science, Mathematics, Art History, Classics, English, languages, Philosophy, Religious Studies, and The Sheehan Art Gallery.

Immersive Projection

Projecting imagery onto walls, floors, and other physical objects allows multiple people to experience a shared virtual world without the need to wear headsets. The lab space has a fixed front-projection screen on the rear wall and two pull-down side screens and roll out floor coverings that allow for up to three walls and floor projection. Click this link to try our Web previs app to plan your scenes before scheduling your video recording session in the lab. Use the labeled HDMI cables on the lab’s computer table to connect your notebook computer’s video output to the desired projector display. The lab has several portable projectors and tripods available for loan along with a portable rear-projection screen.

Students in a Hispanic Studies class imagine Picasso's Guernica in the time of COVID cleaning precautions. One student stands holding a roll of toilet paper. An image of the Guernica painting is projected onto the rear wall. A second student lies exhausted on the floor holding a spray can of disinfectant. Both studies are wearing masks.

Students in Mariana Perez’s Hispanic Studies class imagine Picasso’s Guernica in the time of COVID cleaning precautions.

Motion Capture

Motion capture or performance capture systems track the position and orientation of moving people or props. This motion data stream can be applied to animated virtual avatars, abstract artistic effects, audio effects, and so forth. For example, the position of a tracked prop might be reported as three numbers that express the position along X, Y, and Z axes, where each number is between 0 and 15 feet. You could then apply a function to map position values between 0 to 15 onto colors around the color wheel so that 0 corresponds to yellow, and 15 corresponds to green.

Markerless Tracking

If you need to track moving people who will walk in and perform with no special setup, use one of the lab’s markerless systems including the Microsoft Kinect V2, Kinect Azure, Orbbec Astra, Intel RealSense D455, or WebCam image processing. Click this link to try a live example of human figure tracking from the Google MediaPipe toolkit that requires only a WebCam.

Marker-based Tracking

If you require greater precision or need to track non-humans including physical props, use one of the lab’s marker-based systems. Attach either passive reflective spheres or active LED infrared or visible red light markers to the subject or prop. The discreet infrared (invisible) active LED markers are ideal for tracking multiple unique performers on stage. The lab’s primary system is an 8-camera Vicon Shogun system. For portable motion capture of human motion, use the Shadow strap-on inertial system that packs down into a single briefcase. In the below image, senior art student Eva Coulon holds a rigid cluster of four passive reflective spheres that plays the role of a virtual crayon that can paint in 3D space. The XYZ position data is streamed into the arts effects software TouchDesigner to create visual effects for Eva’s reimagined Harold and the Purple Crayon. Eva used a TouchDesigner node to map the marker position to a wavy warp effect applied to the story’s text that appears projected onto the floor.

Virtual Reality (VR) and Augmented Reality (AR)

The lab has multiple VR and AR headsets and MSI VR One backpack computers for experiencing AR and VR. It is possible to attach tracking markers to the AR/VR headsets and controllers to create shared experiences where multiple participants interact in a shared virtual environment.