Volumetric recording is the quick kick in the ass that VR so desperately needs |Techilink

AUSTIN, Texas – As has become the norm at work, I recently donned a virtual reality headset to go somewhere far, far away. This day was a journey into space, with pit stops on the surface of a moon and a sci-fi style briefing room, to hear about future expeditions to Mars.

I should have been looking out for the virtual planets, solar sail platforms, and plans for human colonies. But I kept turning my attention elsewhere: Buzz Aldrin was standing next to me.

“You can watch Buzz from every angle,” said a demo rep as I wore an HTC Vive VR headset. It was true. I could stand on tiptoe and look at the top of his head, crouch and look at his chin, or step aside and see his face and body from every angle.

This wasn’t the result of 360-degree video tricks, where a distorted panorama could extend around my field of view, nor was Aldrin made of blocky polygons. With every step I took, this VR system restored Aldrin’s appearance, and the image was based on a series of dozens of cameras that captured his image from all angles.

The effect is achieved through a process known as ‘volumetric capture’. Until now, real-time 3D imaging has not put much emphasis on this resource-hungry technique. As tempted as I was to turn my nose up at the gimmicky character, a mind-blowing demo (not the one with Aldrin) changed my tune. Virtual reality offers a specific use case that breathes new life into the concept – and in a way you pull back, VR has good reason to embrace the standard, no matter how patented it is.

Little Buzz!

A start-up image processing company called 8i has been crowing publicly about its volumetric recording system for over a year and has used festivals such as Sundance, film properties such as Mad Max, and stars like Jon Hamm to get VR newbies excited. Put on this headset and look-ee-here, you’re really standing next to other people! Correctly.

As they’ve said during other public demos, company representatives clarified to Ars that the human VR subjects are captured by a multi-camera array in a green-screen space. Lenses are placed above, below and around the subject in question and their movement is limited; you won’t get the walk-and-dance results of, say, a flatter hologram projection like Tupac or Hatsune Miku. Instead, you get a compelling view of a person (with some compression-related spots) working no matter what angle you see them at. Unfortunately, all my curious technical questions – do you use voxels, how do you refresh the image efficiently in a VR screen, do cameras use infrared recording, which GPU attributes are best for this kind of display – were pushed aside.

Instead, I was offered something really weird: a chance to hang out with little Buzz Aldrin.

A representative held up his smartphone, which contained 8i’s “augmented reality” collection of captured actors and celebrities, and pointed his camera across the room. Aldrin looked like he was in the room (with the usual AR vibrations of smartphone detection, just like in pokemon go), and the famous astronaut started yelling, “Get your ass to Mars!” I was asked if I wanted to stand next to Buzz. I objected: how small can you make it?

Minutes later, I posed with a tiny, tiny American hero in the palm of my hands. (The Rick and Morty fanboy in me spent the rest of the day yelling “I’m Tiny Buzz!”)

8i’s reputation for promotion seems to revolve around moments like this – where people talk less about the technology and more about the “ooh neto” effect of a celebrity portrayed in unusual ways. I laughed when I was sent a laggy video of the moment, then figured I’d put the curio in one of our usual conference photo galleries and be done with it.

Then my South By Southwest changed.

New perspective after After lonely

A few days later I ran into Alex Schwartz, the VR evangelist from Owlchemy Studios (creators of the brilliant Task Simulator). His face had the telltale “raccoon eyes” nick from wearing too many VR headsets, and that’s because he had just finished testing and sampling the festival’s VR award nominees as one of those judges. category. Something good? I asked.

Before I even finished my question, Schwartz said two words: After lonely.

This documentary, which later won the festival’s “room-scale film” award, also uses 8i’s volumetric recording technology, but there’s no Jon Hamm, no Hollywood production value. It’s a cold, dark and stuffy experience. And it completely reversed my expectations of VR documentaries.

After lonely plays Kenneth Moore, a former inmate in a Maine penitentiary who ended up at the center of conversations about appalling prison conditions. Moore’s case was particularly serious, as he was subjected to lengthy solitary confinement and inhumane punishment by guards at the “Supermax” facility.

Some documentaries want you to get involved with their human subjects by stitching together a story that inspires empathy. But After lonelyThe subject is quick to tell viewers that he was not a model citizen in prison. He attacked guards. Tried to start a fire.

Instead, the film uses its setting, and your movement within that setting, to tell its incredible story of isolation and captivity. You wake up in the documentary, which currently requires a room-scale VR system, such as the HTC Vive or Oculus Touch, as Moore’s cellmate. The tiny quarters of an average room-scale VR setup corresponds to how small Moore’s solitary confinement cell was in the prison — a room where he spent more than five years during nearly two decades in prison.

Developers confirmed to Ars that they had pre-baked as much convincing real-life detail into the room rendering as possible, especially the lighting and shadow effects, in order to devote more GPU horsepower to Moore’s volumetric presentation. The result is stunning, and not just because it operates at a smooth 90 frames per second refresh. As Moore recalls his most desperate moments in solitude—pulling hair from his arms and head, using his own blood to write lyrics on the walls—you can walk up to him and look at his face, his tattooed hands, his unwieldy frame. .

I realized as I stood next to the prisoner that the scale of VR settings becomes all the more convincing and emotional when a credible human is with you. Later, I stood twisted outside, looking through Moore’s window, and he looked at me and told me that sometimes he stuck a piece of paper to the window. This required guards to barge in and drag him out, he explained, as it was against the rules. It was his way of getting out.

Then a video screen appeared where his window should be, and I was shown a video of a brutal and inhumane punishment he suffered for doing this paper trick once: he was dragged by guards, then stripped naked and tied to a toilet with a gag in his mouth.

The ending wouldn’t be the same on a flat screen

Depressing and heartbreaking, my feelings about this movie aren’t beginning to explain. But that’s largely due to the technology used here, along with: After lonely‘s filmmakers at Emblematic Group (who produced this for PBS’ frontline) really respect what it means to be a viewer within a VR experience.

Theatrical principles are used with incredible effect. Important text always appears in multiple places in the cinemas, so room-scale viewers never have to search for it. Light and darkness do the same to determine where viewers should focus their eyes. The activity is limited so that the act of being in the scene is less about making viewers filmmakers and more about letting them take the time to understand how the setting affected the man who lived in it for so long.

All of this makes the ending so gripping that I have a hard time typing about it, even though it seems so simple and pedestrian on paper: you’re transported to Moore’s current, real-life bedroom. You stand next to the man as he sits on a large bed and explains how hard it has been to make the transition to life outside. In the room, as he begins to explain things, I noticed that, yes, this bedroom is about the size of his cell. Then he says it: this is his home. He does not leave this room. I’ve been in his cell, and in his bedroom, and I realize that his life will forever be more comfortable in these small spaces. Seeing him next to me—his face, his tattoos, his blank stare as he talks to the wall about his life—frame it all in a way that may never leave me.

Moore doesn’t have to list what he does or doesn’t earn, or rattle off statistics or analysis about prison culture, to convince me of anything. I have been with him. I don’t think anyone should ever experience what he’s done.

This is the beginning of a new wave of VR documentaries and journalism: recording human stories in a realistic environment and folks, and share them in such a way that we’re not left with poor, Cardboard-powered 360-degree videos. Those panoramas force us to work, hunt and peck for stories and insights. After lonelyThe directors and crew have clearly worked hard to remove as many barriers as possible to allow the setting and sensation to do the story.

“Show, don’t tell” just got a lot more interesting and I hope 8i carefully opens its platform to filmmakers and creators with similar storytelling aspirations. It’s nice to say that the 8i’s technology can put me side by side with Buzz Aldrin to imagine life on Mars, but I’d rather the technology let me see more life on Earth.

View image by 8i