VR, Data and Embodied Cognition

VR, Data and Embodied Cognition

Evelyn and I recently attended the Body of Knowledge conference at UC Irvine. The gathering was focused on embodied cognition and the arts. Embodied cognition, and here I’m going to steal Wikipedia’s phrasing, “is the theory that many features of human, or other types of, cognition are shaped by aspects of the body beyond the brain.”

Here’s a quick example: Basic level categories. I explain basic level categories and how they are defined by our bodies and senses in my as-of-yet unreleased zine on our project Unscannables:

“Using unaugmented human senses, it’s fairly easy to distinguish, say, a cat from a dog, but it is only with training and detailed knowledge that one can just as readily delineate dogs by breed. Dog is a basic level category. Table is a basic level category. So is bike, chair, airplane, person, tree, fish, book etc. Any category that you can easily represent with a single image is a basic category. A car? Sure, but a vehicle? Not so much. It’s too big a heading, containing too diverse a group of objects, to label with, or be perceived via, a single silhouette. But it’s not just down to images; a basic level category is also something with which humans use one motor schema to interact or operate. Driving any car feels basically the same but move a step back to horse and carriage, and the physical actions used are completely different. What does all this basic level category stuff have in common? It’s all centered around our bodies: sensory perception, mental images, and physical movement.”

But basic level categories are just one example of how cognition is shaped by our bodies in the environment. At Body of Knowledge, people from various fields (art theory, dance, neuroscience, philosophy, music, anthropology etc.) were all presenting on how focusing on the body was influencing their work. The whole conference felt like an attempt to resist the traditional mind/body dualism of western philosophy. No more “I think therefore I am” around here. Instead, these researchers and artists and scientists were breaking down conceptual separations.

The common theme of many of the speakers was that doing stuff in the world with our bodies (goal directed action) was the source, not the by-product, of cognition. That, while we use mental simulation to test actions before we make them, we actually use our bodies and our physical environments as a scaffolding to create thought. Let’s use a game of RushHour, one of my favorites, to get a clearer picture of how this works.


The goal of RushHour is to get the little red car to the exit by moving it and the other cars back and forth. To play you look at the board and mentally model some series of moves that will get the car closer to the exit, but, depending on how complicated the level is, there will be more steps required to win than you can hold in mind. That’s when some “goal-oriented action” comes in. You manipulate the board with your hands as a scaffold, move things around to a new position to off load some of that simulation and getting a new point of view. Then you start the cycle all over again.


I got a lot of new language for the work we do, but, as is often the case at any conference, it felt like something was missing. When I feel like something is missing, it highlights what I value. I want to use David Kirsh’s talk as an example. He showed the results of a study he conducted which showed that marking, a movement strategy dancers use when practicing and thinking through a dance without doing the piece “full out,” improved performance more than either mental simulation of the choreography or full-out performance.

Marking conserves energy, but more importantly, marking is a form of embodied thinking which utilizes body-based abstraction. (Abstraction, again borrowing from wikipedia, means the “process or result of generalization, removal of properties, or distancing of ideas from objects.”) Kirsh used the example of finger dancing, a kind of marking used my Irish step dancers.

So the findings of Kirsh’s study proved what the dancers already knew: that marking improved performance better that either mental visualization of the task or full out performance. And this is where I felt that missing link I mentioned earlier.

Art practice is primary research. It discovers things that no other form can. It discovers what works, like marking. The fact that dancers have been using marking for generations proves its efficacy. Coming in after the fact and collecting data to prove that marking is more effective than other techniques reads as data-based proof being valued as more true than embodied proof. This is problematic. Data-washing embodied practices biases what we learn from them toward what we can measure about them.

(This is by no means a discouragement from using empirical approaches to show the strength of an effect. As Andrea points out, undervaluing research that proves what is “known” contributes to a problem in science today where results that aren’t “notable” or “surprising” are considered unpublishable. I am instead arguing for taking the findings of embodied practices seriously, even without the data to back up their validity.)

And this is where VR comes in. VR is more than head-mounted escapism, it is bringing computation out of its little boxes and onto our bodies. It’s a relationship that gives our embodied cognition direct access to computation, one of the most powerful tools humans have built. (And holy painted pink cow is that exciting!) But it’s also a relationship that comes with giant piles of super useful body tracking data. Data that we can not confuse with the real thing. Heat maps of spherical videos showing that people mostly tend to look at people and movement should not be read as advice to stick to making videos full of people and motion. It only means that’s what we can currently measure about what is currently being made. Don’t get stuck in the measurable.

So, while I was unenthusiastic about some of the findings and methodologies discussed at Body of Knowledge, underneath those differences was a strong common thread. As Simon Penny, the Conference Director, put it: the work of negotiating between disparate established paradigms is the most difficult intellectual work we can do. I have long described my own art practice as making chimeras, things in between things, and it was exciting to find that focus shared by a group whose fields of study are so different from ours.