AR Stethoscope-MetalDetector-xRay-Walkman — visceral relationships with storytelling

What are the types of stories we would hear in an augmented reality? How would they be told to us?

Designing Environments for Social Systems

5 weeks

Collaborators

Gia Marino

Role

Designer, Technologist

Tools

Processing, Arduino, openAI

Full process here ︎︎︎.

Insight

In our dynamic world, we are constantly surrounded by static objects in our lives, that silently observe us. What if these objects could share valuable hidden insights with us?

Approach

The inspiration for this project came from watching a professor interact with a really lo-fi physical computing prototype in an IoT class I was taking. One of my classmates had put together a color sensor connected to a speaker so that depending on the color the sensor piece was pointed at, a different story would play out loud. Because the sensor was still connected to a breadboard, and the speaker was really cheap and quiet, the whole breadboard piece had to be held really close to the object it was sensing, and the speaker had to be held right up to the ear. Watching a professor test her project with its limitations, I realized, this is basically a stethoscope 💡.

I became fascinated with this idea of stethoscopes: a sensing piece and a sound-playing piece physically connected at a limited distance — a device that allowed you to hear into things. Which leads to a few key questions:

What are the stories which are hidden? Who finds them? And how are they found?

We took an approach of building a world where there are stories detailing the deep-time histories of all objects around us. These stories belonged to the Earth herself, and could only be unlocked with a truly special device, through interactions full of care and commitment.

Implementation

We made a stethoscope device with one hearing and one “sensing” part. Once it is approached and first touched, it rapidly changes colors to indicate that it is “alive” and reactive. The sensing part is meant to be moved around along and across objects until it lands on a story. Once it lands on a story, the color starts pulsing to emphasize the life and story within the object. A live-AI-generated story is played out into the hearing piece. As soon as the sensing piece moves away from the “heart” of the object, the story stops along with the pulsing light. No story is ever the same.

The hearing part is shaped like a shell — the physical form of which indicates being held up to your ear, and the semantic form of it representing listening in to shells. The shell metaphor further carries as the you hear the ocean through the shell, the same way you would hear stories of deep time of the world through this device. The sensing part is also reminiscent of a shell, but circular with a flat bottom, to indicate that it is meant to be pressed against things and slid across surfaces.