Green Screening video on the web – a ThreeJS Extension

posted in: webVR | 0

A while ago, I posted about “10 fun and easy things that anyone can make for VR”. The tenth item on that list suggested green screening your a flat video onto your panoramic photo to create a spherical video.

This is easy to do in pre-processing with video editing software on your computer. For example, Emily helped me take a test video in front of a green screen that looked like this:

greentest_1mp4

 

And using Adobe Premiere (and more Emily video editing help), I was able to insert myself a number of times on top of the “Kirby Cove Panorama” fairly easily. You can check out the original panorama on the eleVR Picture Player – it’s the second image.

greentest2mp4

 

But, for a really interesting experience, I wanted to have the green screened video have the potential to be more interactive. Eg. click a button to start the video on top of the current scene, or move the character in the video around. For that, I really wanted to have the compositing be done live on the web rather than pre-processed.

My first thought was to take advantage of the Alpha channel. On the alpha channel it is possible to encode for “alpha transparency” on some video types, including WebM, a standard web video format. Creating a WebM with alpha transparency was a bit more annoying than I would have liked, but the effect looked great… on Chrome, which has supported WebM transparency since 2013. What I hadn’t realized at the time is that Firefox doesn’t currently have any support WebM transparency. So, as soon as I checked out my project on Firefox, I saw my video in an ugly black rectangle instead of properly composited. Since Firefox is the WebVR browser of choice for a lot of people, this was unacceptable.

I had to go with a different option, and my next thought was to basically do the green screening “live” on the web. Someone, I figured, has probably written a ThreeJS extension to do just that. And, well, there were a few people who had created live green-screened videos with ThreeJS and shared their code, but the projects were all one-offs and not easily “droppable” into other projects. That said, by this point I had seen enough that I was pretty sure that I could use the various projects I’d seen to create my own ThreeJS extension for this purpose.

With the THREEx.chromakey extension, available on Github, all you need to do is include one Javascript file in your library and replace your usual material with the ThreeX.ChromaKey material. Then, in your animate loop, you need to update your material to get the latest frame. I actually think it’s easier to use than the standard way of doing video in ThreeJS.

You can check out the video above composited into another project by going to http://hawksley.github.io/panoramaStory/ pressing space bar once, and then using the WASD keys to look around and find the video (sorry, this was an experimental project with no polish). I’m hiding over by my desk.

Panorama_Story

 

Perhaps more usefully, Emily ended up using the THREEx.chromakey extension in one of her HyperLinkSpace projects, water. In that project (also navigable with WASD), all of the “animated gifs” are actually green screened videos, as the animated GIF format is not at all supported by webGL.

hyperlinkspace

 

THREEx.chromakey has been a nice tool to have handy around the eleVR office as we’ve done different experiments. We’re finally releasing it separately on Github today in the hopes that other people might also find it useful for their projects. We look forward to seeing how other people use it.

-Andrea