Expressive Programming in VR

posted in: Tech Posts - Other | 0

It’s a little early to be writing about this project, but the amount of interest it has generated among our colleagues has led to its starting to spread by word of mouth and private demo, so I thought I’d share just a little about it this week.


We’re currently prototyping designs for a programming language that is both visual and embodied, using the full capabilities of VR as a fundamental part of its architecture. It is intended to be intuitive, sculptural, and expressive, taking advantage of special abilities humans have in embodied 3D space: recognition of objects and patterns, scale, environments, motor schema, and the ability to simultaneously manipulate multiple objects in space by hand without having to think too much about it.

We’d like for it to be naturally very good at programming for VR, but much more fundamental to its design is that you’re programming in VR, which we believe will be helpful to anything you want to program, from VR games to pure math functions.

Just to be clear: we have no plans to implement this language at this point, and we assume that the things we’re playing with now will look absolutely nothing like whatever design we end up with (not just graphics-wise, but at the very core of what it means to program). But all that being said, we’ve been mocking up what it might be like to program in an expressive embodied VR language by creating 3d programming blocks and pretending to program with them in virtual reality.

Here’s some example programming blocks, combined into short functions in VR:

Screen Shot 2017-04-12 at 1.13.07 PM

Above, you can see that the fire, when touched, will cause an “ouch” sound effect to be played, and subtract 1 from health. None of this actually works, and the sketching environment we’re working in (Anyland) doesn’t actually have a health functionality, but we’re taking a top-down approach to language design. There’s no reason to work on an implementation when we can imagine it just as well in our heads, and we don’t want to get stuck in a design when we know we’re just starting to scratch the surface on what might be interesting about a VR programming language.

In Anyland, you can write a behavior like “When touched, play bird and emit bluebird,” where “bird” is a stock sound effect and “bluebird” is an object I created and named within Anyland. The first obvious thing to do was to take this text and turn it into a blocks programming language like Scratch (see Evelyn’s post Democratic Design in Anyland, which explores the programming elements of Anyland and compares them to Scratch).

Combining the aesthetics of Scratch and Anyland, it seems like programming elements should be able to be dragged around, copied, saved, and re-used in a way that doesn’t require typing out the behaviors every time. Strangely, the very thing that annoys me about most drag-and-drop blocks-type languages—that it’s easier to just type the thing than to find the right block and click and drag it—is turned on its head in VR, where it’s easy to find blocks in space and grab and move them, but not easy to type.

Very quickly though, we realized that arranging blocks in rectangular sections that snap together from top to bottom, as in a traditional blocks language, was not necessary or natural in VR. We started creating blocks that were more visually identifiable representations of their functions, and we started arranging them in more natural configurations. And rather than having tiny copies of the objects they apply to, we started playing with sticking the code blocks directly onto the original objects in the world.

Soon small bits of code became sculptures, more than the sum of their parts. And that leads to one of the big ideas we’re pursuing: that expressive programming allows for conceptual texture.


1. Conceptual Texture

A certain idea of texture can help the understandability and readability of large programs in traditional programming languages, though not many textural tools are available beyond indentation patterns, comments, and separation of code into different files. The sidebar in Sublime allows the user to see the texture of their program, and given how useful this is despite how terrible it is, I’m excited by how much room there is for improvement. Especially when you have all of 3D space to play with.

We’d like to give the programmer the ability to give their program texture using scale, placement of functions in 3D landscapes, and the ability to express a function in whatever way they feel is right for the job.

Here’s an example of 7 ways to express the same simple program, “when birdhouse is touched, emit bird”:

Screen Shot 2017-04-12 at 1.13.20 PM

The above examples use the stock programming blocks without much modification. An expert programmer might make a more expressive version of the function, to give the function its own cohesive identity rather than being a collection of parts:

Screen Shot 2017-04-14 at 7.31.50 PM

Fluency and skill, for both technical and artistic endeavors, comes from being able to chunk parts into wholes that are more than the sums of their parts. An expert programmer in our imaginary expressive language is able to write functions that have their parts visible, while also being cohesive as a single code-sculpture with a recognizable feel. This recognition of code-chunks as cohesive objects creates a conceptual texture, and helps maintain readability of the larger program.

Screen Shot 2017-04-14 at 7.33.10 PM

Programming language design just hasn’t scaled with the technology that uses them. Basic designs for computer interactions haven’t scaled to fit the extent to which we use computers. We’re still putting files into piles of folders with paths so unmanageable that half the time I find myself going to the internet to re-download the same file that I know I already have somewhere on my computer, simply because the texture of the internet allows me to find things I can’t find on my own computer, where every layer of folders feels the same. And even creating a simple webpage demands piles of folders, libraries, and dependencies that take years of experience to develop a solid mental map of.

But you know what people are really good at? Mapping out thousands and thousands of objects in locations in the real 3D world, even mapping out locations of objects in a place they’ve just visited for the first time. Cities, buildings, floors, rooms of many types, cabinets and shelves, containing boxes and drawers. Hills and forests and trees and creeks with that one fallen tree with the hole by the base with the mushrooms and something living in it. Expressive programming allows conceptual texture, for beautiful programs that can be understood at every scale.


2. Layers of Understanding

We like the idea that a programmed world should have its programming accessible beneath the surface. One idea is to have parallel worlds: the surface world and programming world. When you switch into programming world, the surface world dims and you see it annotated with its programmed behaviors (all editable, of course). Theoretically the language should be able to have parallel worlds below that, all in itself down to assembly.

Regular world:

Screen Shot 2017-04-12 at 1.15.08 PM

One level into the programming world, we see a handful of objects are programmed with behaviors. In the following screenshot, the closest birdhouse has three basic programming blocks attached: “when touched”, “emit”, and “play sound”. The play function references the bird sound effect, and the emit function references a bird object (which is “quoted” by the emit block, rather than functioning here as the bird at full scale). And so, when the birdhouse is touched, it plays a bird sound and a bird flies out.

To the right of the birdhouse, we see that the blades of grass are programmed to play a sound effect and become short when they collide with the lawnmower object, and then grow back over time to full height.
Screen Shot 2017-04-12 at 1.15.25 PM

We should be able to open up the code blocks too, to understand or modify their behaviors. This leads us to another aesthetic decision with huge technical consequences: how deep can you go, and how do the layers change?

On one end of the spectrum, there’s environments that are visual/interactive on the surface, and then by peeling back that layer you’re instantly in javascript or C++. But for this project I’d prefer a smoother transition that allows someone fluent in the VR language to still program/edit in that same language as they look inside the objects, even if deeper layers have less functionality or require more blocks to build up into the higher-level functions. Ideally, it’d be programmed in itself all the way down to the assembly.

3. Turtles All The Way Down

After doing a collection of blocks for Anyland functions and then some more standard language blocks for things like if/else statements, arithmetic, and loops, I wanted to do something different and more complete. Given Seymour Papert’s work on embodied knowledge and programming, LOGO seemed like a good fit. It’s embodied by design, has an obvious and VR-friendly visualization, and is simple/small enough to mock up in its entirety.

The following screenshot shows a turtle almost done running its program to draw a star. If you like, you can run it slowly enough to walk along with it, to literally follow the program.
Screen Shot 2017-04-17 at 10.44.09 PM

The programming blocks are themselves shaped like turtles:

Screen Shot 2017-04-14 at 7.34.40 PM

When you open up a turtle, you can see its program. The program for drawing a star uses the “repeat” function. Rather than using brackets, the repeat function turtle-block uses its arms, bodily embracing its parameters with its tiny turtle arms:

Screen Shot 2017-04-17 at 10.37.50 PM

For greater texture/complexity in larger programs, functions could hug their components close to themselves, to be expanded as needed.

For smaller and less-loopy programs, these blocks have the nice feature that they can be laid out in a way that looks like what they program. Here’s one that writes my name:

Screen Shot 2017-04-17 at 10.36.10 PM

Beyond just creating a blocks-style language but with 3D icons, we want to find ways to make 3D arrangements in space matter. Under this turtle’s shell you can see the loop it uses to draw a square spiral (to try a different arrangement than using the repeat function):

Screen Shot 2017-04-14 at 7.35.22 PM

The blocks animate as they run, for a visual understanding of what the turtle is doing when. To debug, one might need to insert a break point. For this, you can simply turn one of the turtle blocks onto its back, and it will helplessly wiggle its little turtle legs into the air, halting the program.


4. See The Forest For The Trees

Every object contains the objects that program it, which includes chunks like the collision functions that are necessary in order to manipulate the blocks. Here’s the inside-view of one of the turtle programming blocks inside the larger turtle, which includes a grove of trees which contain the collision possibilities:

Screen Shot 2017-04-18 at 9.24.58 PM

From an outside view, it’s visually cohesive as a grove of tree trunks. From an inside view, we can get at the individual IF statements. On the left of the following screenshot is the tree that decides whether to run “if near” or “if walked into”. Starting at the base of the tree, first we check if the object is within a certain distance to our body. If it is, then we do a collision check between the object and our body, to see if we walked into it or are merely near it. Then we can look for where the object’s code uses the “when neared” or “when walked into” block designs, and run the pieces attached to them.

Screen Shot 2017-04-21 at 5.35.36 PM

At a glance, we can see by the texture of the trees whether a collision decision is simple—like whether an object is being pointed at—or complex, like whether an object colliding with a hand is being merely touched, grabbed, shaken, or let go of. We can easily get to the part of the code we care about, rather than scrolling through lines that all look the same. And if I want to change the block that does “when neared” to look like something else (a turtle, perhaps), all I have to do is replace the block on the end of that section of the tree, or add my new one if I want to be able to use both.

In order to get the LOGO blocks working in my VR programming landscape, I need to include the LOGO library as well as the standard library. We can import both libraries, but more importantly, we can go inside them. A good library should not just make its parts available, but lay them out in an organized way that allows the visitor to understand what’s inside, as well as containing its own documentation and help desk.

The Standard Library is a fancy institution designed to house a large amount of information, while the LOGO library with its relatively few blocks more playfully exists underwater, in a lake beneath a turtle-shaped gazebo.

Screen Shot 2017-04-21 at 5.44.54 PM

Inside the standard library:


Inside the LOGO library:


There’s a pile of questions this sort of fiction brings up, and luckily here at HARC we’re surrounded by the right people to help us explore every obscure technical detail of implementation, as well as every out-there idea of how VR interfaces could enable entirely new kinds of programming. Every bit of feedback we get opens a whole new area of intrigue. We’re researching about how to get weird with new kinds of primitives and data types, rethinking what it means to be a variable, rethinking order, maybe even bringing in some machine learning. But first we’re going to work on pretending to program more things in more ways. Recreating LOGO or blocks languages in 3D is about the most boring thing we could possibly do, as far as I’m concerned.

Vi Hart


P.S. If you’d like to pretend to program using these blocks, go to the Public Programming Library area in Anyland to collect them (Anyland is available on Steam for Vive). Better yet, try making your own pretend programming language, and let us know about it! Or better better yet, implement one for real 😀

P.P.S. As mentioned last time, our email address is not currently a thing. Sorry if you’ve tried contacting us there! Try our gmail address, elevirtual.

P.P.P.S. re-linking video tour of some of the work, in case you don’t want to scroll all the way up: