If you’ve seen episodes of the Mandalorian, then you might know that they film live talent in front of a massive curved wall of interlocking LED screens. The LED walls make it possible for regular film cameras to record the talent in front of the virtual background. This reduces production time since visual effects are live and captured by the film camera and the talent can act while seeing the virtual background instead of an empty green screen. I had explored pricing on an LED wall and decided that rear-projection technology gives us much of the same benefits at a far lower cost. This design choice was later validated during SIGGRAH 2021 when I saw Nick Jushchyshyn of Drexel unveil a similar portable rear-projection screen setup.

The temporary basement lab’s rear-projection test used a portable signage frame, gray rear-projection screen from Carl’s Place, a short-throw (0.5:1) projector, and portable LED film lights. Seven of our fourteen Vicon cameras were placed on tripods to provide tracking of the position and aim direction of a camera so the Unreal virtual background would change to match the camera moves.

Many pre-built virtual sets, such as this Stylized Egypt by Aleksandr Ivanov, can be found in the Unreal Engine Marketplace to help you get going quickly. The lab has a diverse collection of pre-built virtual sets from which to choose, or you can build your own virtual sets using Blender or Maya.

The immersive effect is greatly enhanced by tracking the position and orientation of the real film camera, tablet, or smartphone so the virtual background can be continuously re-rendered to reflect camera moves . Students will be able to practice virtual production techniques using a wide-range of tracking technologies from the high-end 14-camera Vicon Shogun system to HTC Vive trackers.

Next – MIDI keyboard makes animated leaves fall