AR Stethoscope-MetalDetector-xRay-Walkman | Visceral relationships with storytelling
GOAL
Focus on the sense of sound to create a novel interpretation of augmented reality.
Focus on the sense of sound to create a novel interpretation of augmented reality.
ABSTRACT
What are the types of stories we would hear in an augmented reality? How would they be told to us?
What are the types of stories we would hear in an augmented reality? How would they be told to us?
Environments Design Studio
Spring 2023
Spring 2023
duration
5 weeks
5 weeks
tools
Processing
Aruduino
OpenAI API
Processing
Aruduino
OpenAI API
role
designer, engineer, fabrication, creative direction, documentation
designer, engineer, fabrication, creative direction, documentation
Full process here.
APPROACH
The inspiration for this project came from watching a professor interact with a really lo-fi physical computing prototype in an IoT class I was taking. One of my classmates had put together a color sensor connected to a speaker so that depending on the color the sensor piece was pointed at, a different story would play out loud. Because the sensor was still connected to a breadboard, and the speaker was really cheap and quiet, the whole breadboard piece had to be held really close to the object it was sensing, and the speaker had to be held right up to the ear. Watching a professor test her project with its limitations, I realized, this is basically a stethoscope 💡.
I became fascinated with this idea of stethoscopes: a sensing piece and a sound-playing piece physically connected at a limited distance — a device that allowed you to hear into things. Which leads to a few key questions:
What are the stories which are hidden? Who finds them? And how are they found?
We took an approach of building a world where there are stories detailing the deep-time histories of all objects around us. These stories belonged to the Earth herself, and could only be unlocked with a truly special device, through interactions full of care and commitment.
IMPLEMENTATION
We made a stethoscope device with one hearing and one “sensing” part. Once it is approached and first touched, it rapidly changes colors to indicate that it is “alive” and reactive. The sensing part is meant to be moved around along and across objects until it lands on a story. Once it lands on a story, the color starts pulsing to emphasize the life and story within the object. A live-AI-generated story is played out into the hearing piece. As soon as the sensing piece moves away from the “heart” of the object, the story stops along with the pulsing light. No story is ever the same.
![]()
The hearing part is shaped like a shell — the physical form of which indicates being held up to your ear, and the semantic form of it representing listening in to shells. The shell metaphor further carries as the you hear the ocean through the shell, the same way you would hear stories of deep time of the world through this device. The sensing part is also reminiscent of a shell, but circular with a flat bottom, to indicate that it is meant to be pressed against things and slid across surfaces.
The hearing part is shaped like a shell — the physical form of which indicates being held up to your ear, and the semantic form of it representing listening in to shells. The shell metaphor further carries as the you hear the ocean through the shell, the same way you would hear stories of deep time of the world through this device. The sensing part is also reminiscent of a shell, but circular with a flat bottom, to indicate that it is meant to be pressed against things and slid across surfaces.
STRATEGIC INSIGHT
Creative direction: we really wanted to push the worldbuilding in the experience design of this technology. Themes we worked with included: terrestrial, the grotesque, childlike wonder, and a contrast between reality/nature and artificial. We wanted the experience to be offputting, but also harmless, to reflect our current hypothesis around AI generated stories.
A lot of the early form inspiration was childhood “magical listening devices”, such as conch shells, cups pressed against the wall, solo cups held together by string. The length between the listening and sensing device is approximately that of a human wingspan; we debated not having the physically connected and having the human torso itself be the connection, but ultimately decided to keep the wire, as it pushed a story of magic — that sound and story is able to travel through string.
Intimate connection to the storytelling experience: This device physically ties you to a story if you wish to hear it, forcing listeners into intimate (sometimes, uncomfortable) positions ie. on hands and knees to listen to something on the ground. There is also no indication of length of the story beforehand, and as the stories are newly generated every time you land back on them, they are completely ephemeral. The volume and form of the device makes it so the story can only be heard by the single listener at a time, further exploring ephemerality in the storytelling experience.
The spectator as part of the experience design: This device is experienced by more than just the listener. This project is also commentary on how we spectate those in our natural world interact with AI and in MR. The volume and form of the device makes it so the story can only be heard by the single listener at a time, and exists in a world of its own — within the world of our own.
Process / GALLERY
