eleVRant: Stereo Polygons

eleVRant: Stereo Polygons

posted in: Camera Research | 0

Last time we talked about camera balls for mono spherical video. Today we’re getting into stereo spherical video, starting with a focus on the most common “3d 360” paradigm: the stereo polygon.

IMG_20140606_185643Full panoramic stereo video is made by capturing two panoramic videos of opposing panoramic twist (see earlier posts to brush up on how that works). You only need two cameras to cover each place where you want to see in stereo, but those two views need to be level with where you expect the viewer’s eyes to be. This is why fancy polyhedral arrangements, as beautiful as they are, are mostly thrown out in favor of discs of cameras that have additional mono coverage on top and bottom, though there are other possibilities.

The simplest arrangements involve one set of cameras for the left eye and one set of cameras for the right eye. You could think of this as being stereo pairs of cameras that then get stitched to other stereo pairs of cameras, which most setups make obvious to see, but it’s more accurate to think of it as two panoramas stitched from views with opposite twist.panocam

The Panocam3D setup is a good canonical example of a stereo polygon, with right/left camera pairs easily visible around a hexagon.

Think of what the left eye sees as it looks around at the left eye video for this camera setup. For anything right in front of you, it always sees the view slightly to the left of it, just as it would in real life. Of course, this holds true even when looking to your left. When you turn your head to the right and look out of the left corner of your eye at that same object in the video, in real life you’d now be seeing the view from the right of that object, but in the video you’re still seeing the same thing you were before. You still see the view from the left.

This makes stereo spherical video fundamentally imperfect, but it’s enough to trick your brain, because whenever you’re looking at an object with both eyes, even if the view isn’t quite realistic, it’s still a consistent stereo view. Your left eye sees lefter than the right eye, and thus you can focus on the object and perceive depth.

Jim Watters' Star Heptagon CamJim Watters’ homemade panoramic setup is a little less obvious, but has the same idea. Seven pairs of cameras around an entire 360 panorama, only the pairs overlap to give a star heptagon rather than an obvious polygon like the panocam.

There’s a twisty circle of cameras for the right eye, and an opposite-twisty circle of cameras for the left. The smaller radius of the setup (compared to a non-star heptagon) should make it easier to stitch.

You can see the cameras in this setup as being in pairs, but the important thing is that right and left views one interpupillary distance apart are parallel and level. Wherever there’s a left-eye view, one IPD to the right is a parallel right-eye view.

We followed this same double-camera-disk idea for our first stereo spherical camera prototype. Four cameras for left eye and four for right eye complete a 360 stereo panorama.

Our foam-core prototype works surprisingly well, though it doesn’t allow for plugging in the cameras (gopro battery life is very unreliable), and doesn’t help GoPro’s overheating problems in hot environments. The camera placement works quite well, though, with all the stereo parts being created by level parallel views from an interpupillary distance apart.

Screen Shot 2014-10-07 at 3.27.00 PM

The Panocam and Jim Watters’ setup shoot a stereo 360 panorama with no top or bottom of the sphere, but with additional cameras their stereo panorama could be completed with a non-stereo top and bottom. There is nothing jarring about a transition from stereo around to mono looking upwards; most people don’t notice the transition, which is why stereo panoramas with mono completion are standard for stereo spherical video right now.

In our prototype above, after the four pairs in a stereo square, four more cameras cover top and bottom, all of which get stitched to both the right and left eye’s video, for a complete spherical video that is stereo around the middle. If gopros had a field of view as tall as it is wide, we’d only need one camera each on top and bottom.

If you’ve been following our previous posts however, you’ll know that if we want stereo on top and bottom, we run into problems. If we have it so there’s a stereo pair of gopros pointing up that stitch such that you can look up from a forward-facing view, then they’ll go out of sync if you look up while turned to the side. You can plan to have your viewer only face their body forwards, or you can plan to have them spin around, but you’re going to have to sacrifice 3d somewhere.

Unusable Camera Mount for 3d SphericalFor an example of what not to do, think of what would happen as you look around a video captured with 360Hero’s flagship rig. It’s designed to be a pretty and polyhedral arrangement of cameras, with no thought given to the result of what those cameras actually capture.

When looking at objects in front of you, sometimes your left eye sees them from slightly to its left as it should, but then, as you turn your head, suddenly you’re seeing the view of an object in front of you as if you were seeing it from slightly above. This alone wouldn’t be a huge problem if the right eye saw the object as seen from slightly above and to the right, but instead the right eye sees the object as seen from slightly below. Instead of level left and right views that mesh to give a stereo image, you get two images that don’t mesh at all unless you suddenly tilt your head 90 degrees.

It’s interesting to think about stereo pairs of spheres designed for viewing with a tilted head or smoothly changing head orientation, but that 360heros rig has huge discontinuities in stereo disparity.

Do not buy this camera holder. I’m still hearing from people wondering why they can’t get their $1000 camera holder to output videos that work, and people still considering buying it because it’s so pretty and well marketed.

Screen Shot 2014-10-07 at 2.26.10 PM

A pyritohedral arrangement, so optimal for mono spherical video, makes zero sense for stereo and so I dismissed the idea on principle, but Emily and Andrea wanted to try it out anyway just to see what it would actually look like, so they hacked together a setup that covers all the relevant singularities and tried stitching a video out of it.

This turned out to be a good idea, because we learned just how sneaky the resulting video is to our brains. Parts of it mesh, and parts of it just barely don’t, and you’re left with a feeling that it almost works, and that maybe with just a little tweaking in the stitching software you could get it to work.

We know mathematically that the error is in the hardware setup and not our stitching, but if we didn’t, we can see how you might think it’s a viable setup and your own fault if you can’t get it to work. It was a really useful exercise in helping us understand the way the brain tricks us and to guard against making similar mistakes.

360heros also has a couple stereo polygon holders for gopros that do work. Gopros arranged vertically have a field of view that will stitch with five cameras around, so there’s two pentagonal arrangements of ten camera disks, one with one gopro each on top and bottom, and one with two on top and two on bottom. 360heros claims that the one with two cameras on top does full stereo all the way around, which as previously mentioned is not mathematically possible. You could have it stereo all the way in a horizontal circle, or theoretically all the way in a vertical circle if you spin upside-down (though you’d have to switch which camera stitches left and which stitches right), but you can’t have both, so I’ve edited their original infographic (left) to reflect what would actually happen if used as suggested (right).

Screen Shot 2014-10-07 at 4.12.50 PM

We got this camera holder for use as standard equatorial 360 with mono top/bottom, and I’m going to review it now:

You don’t actually need both top/bottom cameras to stitch a spherical video that’s fully stereo around the equator and mono on top and bottom, but we thought it would be nice to be able to use both on bottom and stitch around the tripod. It turns out that when the gopros are in the holder they actually cover up the camera holder’s tripod hole. You can mount it on an angle, though the weight makes it difficult to keep level, or leave out one bottom camera. Either way the tripod is quite visible.

IMG_20141007_191015I like that the holder holds the cameras more securely than our friction-fit foam core prototype (securely enough that you need to use a screwdriver for leverage when popping the holders open), but it doesn’t hold them extremely accurately, and there’s often a slight tilt between one camera and its stereo partner. All the buttons and slots are accessible while in the holder so you can keep it plugged in all the time (very important, given inconsistent gopro battery life), though the memory card slot is narrow enough that we need to poke it with a screwdriver through the opening to get the card out. It’s got a lot of flaws, but both Hero360’s pentagonal setups work for equatorial stereo if you know how to ignore their marketing and do it right.

Most importantly, it’s currently available, unlike most 3d camera setups which are still in development or being kickstarted or are proprietary. You could definitely make a better holder yourself for less raw cost, or wait until someone else makes a better one, but for our research group time wins over money and we got pretty much what we expected. I’m hoping someone starts selling a better holder soon, designed using actual theory.

One more example of a stereo polygon: NextVR‘s stereo triangular setup using six RED cameras. A fancy camera with interchangeable lenses means you can stick an extremely wide-angle lens on there, capable of capturing the kind of field of view you need for that sharp an angle between camera pairs. And being REDs, they capture the kind of resolution that still looks good even when you stretch it over a large field of vision.

I’d be curious to see how the stitching looks, and if it’s possible to get anywhere near decent stitching with live capture at that sharp an angle and with camera pairs so far apart. But anyone filming with this camera probably has the resources to do some post-production or non-simultaneous capture tricks to smooth over those errors. The stitching distance on the corners is probably far enough you can’t put anything near the foreground, but on the other hand, far enough that you could hide equipment such as mics and lights that get stitched around completely.

I’ve seen other examples of stereo pair polygons, but I hope this gives you a sense of what’s out there in this space.

So are stereo polygons the way to go? How many sides would we want? Why not just make all the cameras face outwards and grab footage with panoramic twist? Why not do something else entirely?

The short answer to these questions is that fewer cameras means fewer but larger stitching errors, and lower resolution. For a small low-res camera with small field of view, you need seven pairs, but three pairs of REDs sounds good to me. Each of the three stitching locations will be more difficult to stitch, especially if you want anything close or moving between pairs, but it’s probably better to have large safe stitch-free zones if you’re going to have anyone close to the camera.

For video with no forward-facing bias, truly meant to be seen level on the horizon in many directions all around you, stereo polygons plus mono top/bottom are pretty ideal. However, there’s no reason a film needs to stick to one stereo vector field the entire time, and there’s more flexible setups that are possible, in addition to entirely different ideas to explore! We’ll get to that next time.