Monday. Called Eki to catch up. NA. When I got him, he said he'd been on a shoot. That always gets my attention. Asked him what was the project. He said they were doing Virtual Reality tests. And began to tell me how it works. Whoa. I was totally lost.He sent a STAR WARS video of a crew working on VR. I could see a circular screen. Still didn't have a clue. Called him back. He tried to explain. But it was only after I compared it to 1930s pre-on-location movies, where the action might take place in Monte Carlo. The French cafe' is on a set, and the MC background is stock film. But VR surrounds the actors and they can see the action. I think.
He said he'd been working on the technique on and on for three years. And sent a video of the set-up in his home office. Three years sounded like a long time for Eki to nail something. So it must be tough to cracK VR. I want a Dummy: VIRTUAL REALITY 101. Over to you EKI.
Source: personal experience
Next week: FLYING through COVID
First of all, it's not Virtual Reality, but rather Virtual Production. The two are related but not the same. Virtual Reality is when one puts on VR glasses and is immersed in a video game. Virtual Production is when similar techniques are used to produce movies or other video content.
|King Kong, RKO Radio Pictures 1933|
The challenge with this method is that the angle of view is fixed. There is no depth to the background image, which is fine if the screen is used just as an abstract set piece like in television shows, but becomes a problem if the backing is supposed to be a real environment like in movies. The illusion breaks the moment the camera starts moving as the backing is just a flat 2D image - the perspective works correctly just from one single angle.
The solution is to use a computer-generated 3D background that takes the changes in perspective into account. The location and rotation of the real-world camera are tracked in real-time and the background seen on the screen is always rendered from that point of view. When the camera moves, the backdrop changes accordingly, and everything lines up. One can think of it as if the camera was a character in a 3D game - which is actually quite close to how the effect is really done: the backgrounds are usually made with the same game engines that are used when making computer games.
On the high end of the production scale, the video screens are not just flat panels, but rather surround the set and the actors almost 360 degrees, often called a "volume". Even the ceiling can be a screen. This has the benefit of immersing the actors in the scene. The set looks and feels real to those in it - if there's a distant city on the horizon, it's not only seen by the camera but also by the actors and crew on the set. Another important benefit is that the screens act as a light source, giving a realistic ambient light to the set. They also show in reflections, which can be very important if the subject that's being shot is e.g. a car, most of the look of car paint actually comes from the reflected environment.
|Lucia – a Christmas story|
On television, real-time virtual sets have been used since the late 1990s but they required state-of-the-art supercomputers of the day and were rather cost prohibitive. In Finland, only the large television broadcasters could afford setups like this. They were also horribly non-user-friendly. I worked with one of these systems back in the day, created the virtual sets for a digital sports channel in 1999 or so, and it was quite frustrating, to say the least.
|The green screen stage at SW5 Studio, freshly painted.|
The virtual background is created using Unreal Engine. Unreal is a popular and very powerful piece of software, it can be used to create first-class video games - and the photorealistic virtual sets we need. Best of all, it's free. The data from the Vive tracker drives a virtual camera inside a "video game". The position, orientation, and lens properties are matched as closely as possible to the real camera.