Today we discuss panoramic 3d video capture and how understanding its geometry leads to some new potential focus techniques.
With ordinary 2-camera stereoscopy, like you see at a 3d movie, each camera captures its own partial panorama of video, so the two partial circles of video are part of two side-by-side panoramas, each centering on a different point (where the cameras are).
This is great if you want to stare straight ahead from a fixed position. The eyes can measure the depth of any object in the middle of this Venn diagram of overlap. I think of the line of sight as being vectors shooting out of your eyeballs, and when those vectors hit an object from different angles, you get 3d information. When something’s closer, the vectors hit at a wider angle, and when an object is really far away, the vectors approach being parallel.
But even if both these cameras captured spherically, you’d have problems once you turn your head. Your ability to measure depth lessens and lessens, with generally smaller vector angles, until when you’re staring directly to the right they overlap entirely, zero angle no matter how close or far something is. And when you turn to face behind you, the panoramas are backwards, in a way that makes it impossible to focus your eyes on anything.
So a setup with two separate 360 panoramas captured an eye-width apart is no good for actual panoramas. But you can stitch together a panorama using pairs of cameras an eye-width apart, where the center of the panorama is not on any one camera but at the center of a ball of cameras. What does the Venn diagram of their final panoramic video capture look like?
Each set of cameras is centered around the same point, so the circles overlap entirely. It’s the sort of thing that gave me a moment of: wait, how can this possibly work? How do two panoramas around the same point actually give a stereo effect? But the panoramas are not the same, the footage is definitely different, offset somehow, and it’s more than just turning the footage. So, what is the relationship between space and these two different circles?
Here’s where thinking of eyes as vectors really helps. When you put the vectors of camera direction on the circle for each eye, you can see they capture space with a twist, and you’ve got one circle of each chirality.
Depending on the field of view that gets captured and how it’s stitched together, a four-cameras-per-eye setup might produce something with more or less twist, and more or less twist-reduction between cameras. Ideally, you’d have a many camera setup that lets you get a fully symmetric twist around each panorama.
Or, for a circle of lots of cameras facing directly outward, you could crop the footage for each camera: stitch together the right parts of each camera’s capture for the left eye, and the left parts of each camera’s capture for the right eye.
But just how much to crop? What is the ideal twist angle?
Basically you need some ratio between the radius of the circle of cameras (which in this first case we’ll assume is the circle your head would turn in while staying still) and the distance between your eyes. for me, this might be about 8cm for radius from center of head to eyes, and 6cm for interpupillary (between-pupils) distance. So you take a circle of radius 8, and a chord of distance 6. Your two eyes shoot out a pair of parallel rays staring straight ahead, and you want to know the angle between the right eye shooting out of the sphere parallel to the left ray, and the ray that would shoot straight out from the circle.
That’s is all the information you need to find the angle by doing geometry to it, and then you’d figure out the field of view to pull from each camera to get the footage from that angle and stitch it together into a panorama with a twist.
But I’m less interested in the details of that than the much more interesting question: what happens if you make the angle different than is natural?
Imagine angling the field of view of two cameras towards each other. Now, when you look straight out ahead with parallel eye rays, the cameras make those rays meet at some finite distance. When you look closer, the cameras make it seem like you’re looking even closer. All of space moves inward.
While if you angle the cameras out, the world moves further away, until it’s all flat at infinity, and after that you won’t be able to focus on anything.
This suggests an interesting way to focus a panoramic camera closer or further. The camera stays still and the field of view stays the same, but by slowly shifting what footage in your multi-camera-panorama gets stitched into the final twisty panorama, theoretically you could “zoom in” and make everything seem closer.
Even more interestingly, once you have the footage, there’s more subtle twisting you can do in post-production.
I like to think of the twisty-circle as a vector field. Vector fields on the circle behave so nicely, with many possible continuous vector fields to choose from! What if the angles of footage pulled for stitching changed depending on what part of the circle you’re on? You could make certain sections of your panorama seem closer or further, and with enough cameras, even focus on individual objects by giving them the stereoscopic illusion of sticking out. All with regular video filmed with regular cameras, and exported into two single flat panoramic videos.
But what about spherical?
Spheres are weird, and vector fields on spheres are weird, and twisting 3-space to collapse it onto the sphere is weird. Maybe more about that next time, or maybe we’ll put together some examples using these different techniques. Who knows where we’ll go! Wooo, research!