Now that the Virtual Venn Diagram Museum is a thing (see my previous post), we’re working on bringing some of the exhibits into the real world. This is in part because it’s the obvious thing to do for some of them, showing VR’s power as a sketching tool for reality. It’s also partly to mix up my thinking about interactive media and the design process.
I’ve been working on prototyping about 20 different exhibits. Here’s some of the ideas:
This week I decided to go deeply into one exhibit, the Venn Piano.
In the virtual museum’s music exhibit there are four Venn Pianos, each an interactive explanation of one of the four chords at the foundation of Western music theory: major, minor, augmented, and diminished. By touching the different sections of the diagram, you hear the corresponding combination of notes.
I’d love a real acoustic instrument that does this, but playing two or three notes at once is much easier in the virtual world. So I decided to make a digital instrument where each section of the diagram senses when it is touched by the mallet (or your hand) using capacitive sensing.
I traced a plate onto aluminum foil into a Venn Diagram, cut out the sections and wired them over to a controller.
I used the MakeyMakey, which makes it super simple to turn capacitive input into keyboard strokes on a computer.
To make the correct notes play according to different keyboard input, I used Scratch, because it seemed right to use a spatial blocks programming language for things related to research into embodied understanding. Because it’s a very visual language designed for visual things, it was easy to incidentally create an ugly Venn Piano that you can play on your screen (if you’re on a computer):
You can edit and remix on the Scratch project page, if you want.
I arranged the main control blocks into two overlapping venn diagrams, to help me keep track of which is which. Now I’m thinking it would be nice if programming languages let you sketch on them, so that I could draw in the circles and label them as a sort of “comment”, like this:
Interestingly, it was much easier to create this project in Anyland in VR than to use Scratch to control the MakeyMakey. That’s because in the virtual world, the lack of real collision physics allowed me to simplify the design. Instead of having to create seven different sections for each Venn piano, each with their own collision, triggers, and output, I could truly just make three circles and overlap them. When you touch the sections, your virtual hand goes right through the virtual objects. Touch the middle section and you intersect through all three intersecting circles, and each can simply play its own note.
This is something I’d like to keep in mind for our group’s future programming language research. Maybe visual/spatial programming tools could take advantage of the way non-physical objects can overlap and intersect. What if I had big programming blocks that I could overlap to generate a new but predictable functionality?
Anyway, after the physical version existed and I showed it to some folks, I thought that maybe I should make some more material that helps explain the concept behind the piano to people without a background in music theory. I started by putting the three notes in a more familiar context: a toy xylophone with all the other notes taken out. I annotated the physical xylophone to show the different combinations of notes, and drew a Venn diagram that uses traditional music notation, too. And just for good measure, I tuned a 3-stringed ukelele to a C major chord too.
Writing out the music notation brought me back to when I was first studying music theory in high school and spent hours and hours filling out worksheets of chords, scales, and intervals. Though at first the class dreaded these packets of worksheets, we grew to love them, as the effects on our skills was undeniable. I decided to create a companion worksheet for the exhibit, which I hope you’ll try out if you’re working on music theory yourself. It’s available as a PDF or as the following image:
I’ve been thinking a lot about worksheets, lately. They’re a classic explanatory medium, designed for certain kinds of basic interactions. They’re also a set of instructions that can include directions to do things outside of the worksheet. I’m going to experiment with making more worksheets that connect with the physical and virtual exhibits, to see if adding this medium to my thought process can help me design and explain better.
But still, the above worksheet is designed for someone who has started learning music theory already, not someone with zero music theory background. So I made a video that shows the different virtual and real variations of the Venn Piano, plus explains some basic music theory using a real piano:
Finally, I took the video and worksheet and brought them back into the virtual exhibit. Someone without enough music theory background to understand the interactive virtual exhibit can now watch the video while playing with the Venn Pianos, and someone interested in taking their knowledge further can pick up a worksheet.
The video and worksheets, being pure information, work in the virtual exhibit as well as with the physical version. As for the Venn Pianos, I actually prefer the virtual ones! It feels more natural to play in VR using my virtual hands than to either use or keep hold of a grounded element in the capacitive version. I think if it were acoustic, especially if it had tactile subtleties such as volume, I think I’d like a real version better. But a flat digital version has nothing over a VR version to me. In addition, the VR version took minutes instead of hours, and had zero material cost.
Now the idea of representing chords in a Venn Diagram seems obvious to me, but I’ve never heard of it before, and probably wouldn’t have thought of it without spending so much time designing virtual exhibits. Out of all the possible things one could create in Anyland, the design for a Venn Piano is close to the surface, so to speak. It doesn’t take very many moves to get there. I believe future VR programming/creation interfaces will put an incredible range of new things close enough for us to reach them.
I’ll write up more higher-level insights after I work and write about more physical exhibits. Next time I’ll talk about some exhibits that really shine when they’re in the real world!
P.S. Our elevr email account has apparently been broken for months, so my apologies if you’ve tried to contact us. Try resending your email to our gmail address, “elevirtual”.
P.P.S. The virtual museum is, as ever, playable in the “Venn Diagram” area in Anyland, available on Steam for Vive.