What is Virtual Production?

Virtual production is the conceptualization and creation of a 3D virtual world in a real-world environment. Virtual production has been around for years and continues to grow in popularity as the production process shifts towards a more flexible remote work structure. 

Building virtual worlds

When VR first came out, a lot of animators were simultaneously excited about the possibilities and hesitant to invest in such a new technology. But with the changes brought about by COVID-19 and Facebook’s big commitment to creating the metaverse, it’s clear that the virtual world is here to stay. But will we ever create a virtual world that’s at “Ready Player One” levels? Rokoko’s chief of development says: “art has always led technology forward. Scientists have long drawn inspiration from artists’ imaginings. You just have to look at movies like Tron, Ready Player One, and even The Valerian to know the future’s on its way”.

Virtual production for the film industry using an LED wall

Epic Games and Industrial Light & Magic (ILM) pooled their collective resources to create the Mandalorian. But to bring the Star Wars vista to life on a TV series, they stepped away from the usual green screen, and looked at how they could do more on the physical set. The Mandalorian set is completely encased by a gigantic curved wall of LED screens that allow game engine real-time rendering and video wall technology to come together and create a backdrop for the big beautiful world of Star Wars. 

The screen itself is 21 ft tall and 71 ft in diameter. It’s run by 7 machines that work to comp the visuals onto the screen in real-time. The visuals are created in pre-production and once finalized, can be on the screen in 24 hours. One of the most impressive aspects is that the director of photography can easily move a mountain or landmark in the virtual environment without impacting the real time render. 

The screen is so big, that the production crew says it encompasses your peripheral vision entirely. Unlike green screens, it allows the actors to truly step into the world of the scene, and can accurately interact with 3D characters or props in camera. It meant that when the DP was controlling camera movement, he could instantly understand the correct camera angles in relation to the live actors. 

Virtual production for live events & video games 

Virtual event production is becoming increasingly popular. The idea of using game engines has largely been pioneered by Deadmau5, a DJ who creates imagination-defying AR light shows at his live events. Deadmau5 uses game engines to build virtual sets, plan his performances, and render effects in real time. Fully virtual events are also picking up steam, with Fortnite holding virtual concerts that have been immaculately planned out by Epic Games’ production team. These events draw record numbers of players, reaching up to 12.3 million concurrent users for the “Astronomical” concert. 

What software is used for virtual productions?

In the interview below with Matt Workman (a specialist in virtual production), you’ll learn that Unreal Engine is now the virtual production software of choice. Typically considered in the category of real-time game engines, it’s now moving into the film industry and was the software used to develop the Mandalorian’s graphics. An interview with Matt Workman on virtual filmmaking:

The virtual production pipeline doesn’t have to be set up for these mega-events. You can set up a virtual production studio with minimal budget and effort at home. Matt goes on to say you can use the same virtual production techniques he used on the Mandalorian’s virtual set by using Unreal Engine for free, and a few small screens at home. 

Mixing virtual production with 3D avatars

So far, we’ve covered the production process where the main actors were largely filmed in real life, but it’s also possible to create everything in the 3D world using mocap and animated avatars. Hypothetically, this means that you could ‘shoot’ a 3D animated film in live-action! The National Film School of Denmark’s students use Rokoko’s motion capture software and suit to create animated short films in real-time:

Virtual production studio setups

An indie studio setup doesn’t have to be expensive. Here’s what you need to get started:

• A mocap suit, like the Smartsuit Pro
• Actors (or just yourself) 
• Someone to control virtual cameras via a tablet
• HTC trackers
• Space to record
• Props of your 3D objects
• Tape to mark the actor’s ‘virtual’ space
• Any additional mocap tools for facial and hand motion capture, like Rokoko Face Capture and Smartgloves
• A computer loaded with Unreal Engine and Rokoko Studio
• A digital set 
• Digital assets (eg. characters, environments, audio) 

How to get a job in virtual production

Want to get in on the future of film production? As Matt Workman says, virtual productions use the same visual effects supervisors, directors, and local teams of people on set. With around 300 LED screens in action worldwide, the process to get a job in the virtual production environment is pretty similar to other film-related jobs. You just need to make it onto the right physical production. 

A strong showreel, working knowledge of Unreal Engine, experience with mocap, and a keen understanding of machine learning are all skills that are likely to put your CV at the top of the pile. Making sure you understand traditional production, creative workflows, and how to bring digital worlds to life, will also work in your favor. 

You can get work as a virtual production supervisor, virtual scouting, executive producer, vfx supervisor, head of performance capture, or any number of creative roles. Virtual Production is the new playground for content creators. A few years ago, something like virtual reality used to be completely out of reach for the indie studios of individual creators. But now you can create a virtual production with minimal investment into new tools. Want to get started with virtual production? 

• Learn how the industry is using Unreal Engine to power the production of virtual events 
• See how to get started with virtual production using Rokoko’s mocap tools
• An extremely detailed video walkthrough of filming and compositing live action on a virtual set for a music video 
• How to teach virtual production in the classroom (without massive budgets) hosted by Unreal Engine and expert Matt Workman