Evelyn Eastmond, M Eifler, Vi Hart, Andrea Hawksley
2016 – Ongoing
A project to understand how VR/AR technology can extend human ability to program computers.
Natural Language and VR Programming, how GuriVR uses natural language (including English and Spanish) to program in VR.
Democratic Design in Anyland, which explores the programming elements of Anyland and compares them to Scratch.
We’re currently prototyping designs for a VR programming interface/language that is both visual and embodied. It is intended to be intuitive, sculptural, and expressive, taking advantage of special abilities humans have in embodied 3D space: recognition of objects and patterns, scale, environments, and motor schema.
We’ve been mocking up what it might be like to program in this sort of language, using pieces created in Anyland.
Here’s some example programming blocks, combined into short functions in VR:
1. The Code Beneath
We like the idea that a programmed world should have its programming accessible beneath the surface. Following the aesthetic of smalltalk and lively, the program-world should also be programmed in itself, so that you can see the code running the code, all the way down. One idea is to have parallel worlds: the surface world and programming world. When you switch into programming world, the surface world dims and you see it annotated with its programmed behaviors (all editable, of course). Theoretically the language should be able to have parallel worlds below that, all in itself down to assembly.
One level into the programming world, we see a handful of objects are programmed with behaviors. In the following screenshot, the closest birdhouse has three basic programming blocks attached: “when touched”, “emit”, and “play sound”. The play function references the bird sound effect, and the emit function references a bird object (which is “quoted” by the emit block, rather than functioning here as the bird at full scale). And so, when the birdhouse is touched, it plays a bird sound and a bird flies out.
To the right, we see that the blades of grass are programmed to play a sound effect and become short when they collide with the lawnmower object, and then grow back over time to full height.
2. LOGO Turtles
VR programming blocks for a LOGO turtle:
When you open up a turtle, you can see its program. Here’s one that writes my name:
Beyond just creating a blocks-style language but with 3D icons, we want to find ways to make 3D arrangements in space matter. Under this turtle’s shell you can see the loop it uses to draw a square spiral:
The following loop for a star uses the “repeat” function. Rather than using brackets, the function bodily embraces its parameters with its tiny turtle arms:
3. An Expressive Language
A certain idea of texture can help the understandability and readability of large programs in traditional programming languages, though not many textural tools are available beyond indentation patterns, comments, and separation of code into different files. We’d like to give the programmer the ability to give their program texture using scale, placement of functions in 3D landscapes, and the ability to express a function in whatever way they feel is right for the job.
Here’s an example of 7 ways to express the same simple program, “when birdhouse is touched, emit bird”:
The above examples use the stock programming blocks without much modification. An expert programmer might make a more expressive version of the function, to give the function its own cohesive identity rather than being a collection of parts:
Fluency and skill, for both technical and artistic endeavors, comes from being able to chunk parts into wholes that are more than the sums of their parts. An expert programmer in an expressive language is able to write functions that have their parts visible, while also being cohesive as a single code-sculpture with a recognizable feel. This recognition of code-chunks as cohesive objects creates a “conceptual texture”, and helps maintain readability of the larger program.
Expressive programming allows conceptual texture, for beautiful programs that can be understood at every scale.
4. See The Forest For The Trees
Every object contains the objects that program it, which includes chunks like the collision functions. In the world I’m working in, if something is a collidable object that acts on the standard kinds of body collisions, I can include a grove of trees which contain the collision possibilities:
From an outside view, it’s cohesive as a grove of tree trunks. From an inside view, we can get at the individual IF statements. Here’s part of the tree that decides whether to run “if near” or “if walked into”:
At a glance, we can see by the texture of the trees whether a collision decision is simple—like whether an object is being looked at—or complex, like whether an object colliding with a hand is being merely touched, grabbed, shaken, or let go of.
5. Human-Compiled Examples
We think the biggest limitation to computing is not what can theoretically be programmed now, but our ability to understand and manipulate systems of extremely large code bases with complex algorithms and dependencies. Prioritizing understandability over implementation, we’re creating “human-compiled” example programs that radically reimagine how we might program in a spatial, embodied, non-linear, hands-on way.
In our human-compiled program designs, there’s no such thing as a syntax error. If you understand it, it works. Our goal is to increase the complexity of what a human compiler can understand.
Calling Fibonacci Dog, re: Variable Frog
Video tour of some of the work:
What would it look like for these VR programs to co-exist with objects in the real world?
To begin exploring this question, we have exported 8 of the basic programming blocks from our VR programming language into a freely available webAR app, using fiducial markers for each block. We want to see how it feels to have coding available to you wherever you go, and if the ubiquity inspires more situated, embodied ways to think of programming:
The 8 AR Programming blocks: when touched, touch, loop, say, hear, if/else, look, increase health
AR Program: when ukulele touched > hear music
AR program: if dinner is ready > say ‘Dinner’s Ready!’