Film from a single point, then wander around inside a cloud of pixels in 3D

People wearing 360° spherical video headsets will get a feeling of presence when the small subconscious movements they make are reflected in what they say. This is the first aim of Six Degrees of Freedom video (6DoF). The scene changes as the viewer turns in three axes and moves in three axes. 6DoF video is stored as a sphere of pixels and a channel of information that defines how far each of those pixels are from the camera.

Josh Gladstone has been experimenting with creating point clouds of pixels. His 4th video in a series about working on a sphere of pixels plus depth shows him wondering around a 3D environment that was captured by filming from a single point.

The scenes he uses in his series were filmed on an GoPro Odyssey camera. The footage recorded by its 16 sensors was then processed by the Google Jump online service to produce a sphere of pixels plus a depth map.

The pixels that are closest to the camera have the brighter corresponding pixels in the depth map.

360° spherical video point clouds are made up of a sphere of pixels whose distance from the centre point have been modified based on a depth map.

Josh has written scripts in Unity – a 3D game development environment – that allow real-time rendering of these point clouds. Real time is important because users will expect VR headsets to be able to render in real time as they turn their heads and move around inside virtual spaces.

You can move around inside this cloud of pixels filmed from a single point:

In the latest video in his series Josh Gladstone simulates how a VR headset can be used to move around inside point clouds generated from information captured by 360° spherical video camera rigs. He also shows how combining multiple point clouds based on video taken from multiple positions could be the basis of recording full 3D environments:

What starts as an experiment in a 3D game engine is destined to be in post production applications like Apple’s Motion 5 and Adobe After Effects, and maybe eventually in NLEs like Final Cut Pro X.

I’m looking forward to playing around inside point clouds.