Evelyn Eastmond, M Eifler, Vi Hart, Andrea Hawksley

2016 – Ongoing

A project to understand how VR/AR technology can extend human ability to program computers.

Related posts:

Natural Language and VR Programming, how GuriVR uses natural language (including English and Spanish) to program in VR.

Democratic Design in Anyland, which explores the programming elements of Anyland and compares them to Scratch.


We’re currently prototyping designs for a VR programming interface/language that is both visual and embodied. It is intended to be intuitive, sculptural, and expressive, taking advantage of special abilities humans have in embodied 3D space: recognition of objects and patterns, scale, environments, and motor schema.

We’ve been mocking up what it might be like to program in this sort of language, using pieces created in Anyland.

Here’s some example programming blocks, combined into short functions in VR:

Screen Shot 2017-04-12 at 1.13.07 PM


1. The Code Beneath

We like the idea that a programmed world should have its programming accessible beneath the surface. Following the aesthetic of smalltalk and lively, the program-world should also be programmed in itself, so that you can see the code running the code, all the way down. One idea is to have parallel worlds: the surface world and programming world. When you switch into programming world, the surface world dims and you see it annotated with its programmed behaviors (all editable, of course). Theoretically the language should be able to have parallel worlds below that, all in itself down to assembly.

Regular world:

Screen Shot 2017-04-12 at 1.15.08 PM

One level into the programming world, we see a handful of objects are programmed with behaviors. In the following screenshot, the closest birdhouse has three basic programming blocks attached: “when touched”, “emit”, and “play sound”. The play function references the bird sound effect, and the emit function references a bird object (which is “quoted” by the emit block, rather than functioning here as the bird at full scale). And so, when the birdhouse is touched, it plays a bird sound and a bird flies out.

To the right, we see that the blades of grass are programmed to play a sound effect and become short when they collide with the lawnmower object, and then grow back over time to full height.
Screen Shot 2017-04-12 at 1.15.25 PM


2. LOGO Turtles

VR programming blocks for a LOGO turtle:

Screen Shot 2017-04-14 at 7.34.40 PM

When you open up a turtle, you can see its program. Here’s one that writes my name:

Screen Shot 2017-04-17 at 10.36.10 PM

Beyond just creating a blocks-style language but with 3D icons, we want to find ways to make 3D arrangements in space matter. Under this turtle’s shell you can see the loop it uses to draw a square spiral:

Screen Shot 2017-04-14 at 7.35.22 PM

The following loop for a star uses the “repeat” function. Rather than using brackets, the function bodily embraces its parameters with its tiny turtle arms:

Screen Shot 2017-04-17 at 10.37.50 PM


Screen Shot 2017-04-17 at 10.44.09 PM


3. An Expressive Language

A certain idea of texture can help the understandability and readability of large programs in traditional programming languages, though not many textural tools are available beyond indentation patterns, comments, and separation of code into different files. We’d like to give the programmer the ability to give their program texture using scale, placement of functions in 3D landscapes, and the ability to express a function in whatever way they feel is right for the job.

Here’s an example of 7 ways to express the same simple program, “when birdhouse is touched, emit bird”:

Screen Shot 2017-04-12 at 1.13.20 PM

The above examples use the stock programming blocks without much modification. An expert programmer might make a more expressive version of the function, to give the function its own cohesive identity rather than being a collection of parts:

Screen Shot 2017-04-14 at 7.31.50 PM

Fluency and skill, for both technical and artistic endeavors, comes from being able to chunk parts into wholes that are more than the sums of their parts. An expert programmer in an expressive language is able to write functions that have their parts visible, while also being cohesive as a single code-sculpture with a recognizable feel. This recognition of code-chunks as cohesive objects creates a “conceptual texture”, and helps maintain readability of the larger program.

Screen Shot 2017-04-14 at 7.33.10 PM

Expressive programming allows conceptual texture, for beautiful programs that can be understood at every scale.

4. See The Forest For The Trees

Every object contains the objects that program it, which includes chunks like the collision functions. In the world I’m working in, if something is a collidable object that acts on the standard kinds of body collisions, I can include a grove of trees which contain the collision possibilities:

Screen Shot 2017-04-18 at 9.24.58 PM

From an outside view, it’s cohesive as a grove of tree trunks. From an inside view, we can get at the individual IF statements. Here’s part of the tree that decides whether to run “if near” or “if walked into”:

Screen Shot 2017-04-18 at 9.24.25 PM

At a glance, we can see by the texture of the trees whether a collision decision is simple—like whether an object is being looked at—or complex, like whether an object colliding with a hand is being merely touched, grabbed, shaken, or let go of.

Screen Shot 2017-04-18 at 9.25.07 PM

5. Human-Compiled Examples

We think the biggest limitation to computing is not what can theoretically be programmed now, but our ability to understand and manipulate systems of extremely large code bases with complex algorithms and dependencies. Prioritizing understandability over implementation, we’re creating “human-compiled” example programs that radically reimagine how we might program in a spatial, embodied, non-linear, hands-on way.

In our human-compiled program designs, there’s no such thing as a syntax error. If you understand it, it works. Our goal is to increase the complexity of what a human compiler can understand.


Calling Fibonacci Dog, re: Variable Frog

Screen Shot 2017-06-08 at 5.00.51 PMScreen Shot 2017-06-08 at 4.59.57 PM

Screen Shot 2017-06-08 at 4.59.29 PM


Neural Nets:






Video tour of some of the work:

6. Embodied VR Programming Out in the World

What would it look like for these VR programs to co-exist with objects in the real world?
To begin exploring this question, we have exported 8 of the basic programming blocks from our VR programming language into a freely available webAR app, using fiducial markers for each block. We want to see how it feels to have coding available to you wherever you go, and if the ubiquity inspires more situated, embodied ways to think of programming:

ar-programming-blocksThe 8 AR Programming blocks: when touched, touch, loop, say, hear, if/else, look, increase health

Click to Launch

Remix on Glitch


AR Program: when ukulele touched > hear music


AR program: if dinner is ready > say ‘Dinner’s Ready!’