The Sounds of Jello

When I started grad school I never expected to become an expert in... Jello. But here we are, with me known as the resident Jello expert at NYU ITP. I've turned this new obsession into an ongoing collaboration with a Jello company in Brooklyn and plan to continue exploring what I can do with unconventional materials!

Jello Proof of Concept

Last semester I set out to create a gelatinous user interface after I learned that Jello is a conductive material. While gelatin itself is an electrical insulator, the water in solidified jello makes it conductive. A human finger is also somewhat conductive due to water in the skin. A capacitive touch sensor can detect when two conductive objects touch by even the slightest amount. This meant that I could turn Jello into a functional button by sticking a wire in it, plugging the wire in to a capacitive touch sensor, and processing the sensor data. I tested this theory by wiring up some Jello and writing a short program to turn on/off an LED when I poked the Jello. It worked!

Turning on/off the red light in the bottom left corner via touching Jello

After unlocking the Jello button behavior, I decided to design and build a synthesizer to highlight the material properties of gelatin. Gelatin has a soft, squishy, and wiggly feel and I wanted to design a physical interface that could produce sounds matching that aesthetic. DJs work with controllers and drum pads that have lots of buttons, knobs, and sliders and I wanted to mimic all of that with Jello.

It was very easy to get a proof of concept working. Using the Jello capacitive touch sensor setup, I used the sensor data to generate MIDI commands - numerical codes that music-making software can process into different note pitches. I processed the MIDI commands in Ableton Live and triggered some Jello-y sounds to play in response to these commands. So Jello → Arduino code → MIDI commands → Ableton → sound effects.

Touching the Jello for different lengths of time affects the pitch of the sounds produced

DJELLO

And thus, DJELLO was born. I spent the rest of the semester polishing this into an aesthetic interface with a slick acrylic housing. More details and documentation about my design and fabrication process can be found here. I'm very proud of how this project turned out given that this was my first major fabrication project at ITP. I became very familiar with Canal Plastics and using laser cutters.

DJELLO promotional banner

DJELLO demo video

This was also my first opportunity to use ITP's amazing Documentation Lab services to document my project. They helped me capture my project in all its glory before I had to dismantle it to store it over winter break. I'm not used to working on projects that will only exist for a short period of time, since most of my work has been digital. The photos and footage I received from the Doc Lab were perfect for creating promotional material for DJELLO.

Doc Lab test footage

ITP Winter Show

I showed DJELLO at the ITP Winter Show '23, armed with several tupperware containers full of backup Jello... Jello cubes, glitter Jello, rainbow Jello, the works. And it's a good thing I did, because DJELLO was very popular with the kids, some of whom were NOT gentle. There was also one grown man who came by, took one look at the Jello, and immediately slapped it with his whole open palm, causing it to splatter everywhere.

I learned a lot from having the public interact with my work. What people found surprising, what they found intuitive, what delighted the most. Someone also described DJELLO as an "ephemeral interface," which is a good description that I hadn't considered!

Rainbow Jello, made by pouring one colored layer at a time, several hours apart

Showtime! I had an ice pack under the acrylic to keep the Jello solid for several hours

Kids interacting nicely with DJELLO

Solid Wiggles Collaboration

During the show, someone interacted with my work and told me that his friends own a boutique Jello treat company, Solid Wiggles, and could be interested in my work! I later reached out to them about a potential collaboration, to combine my gelatin interface tech with their beautiful Jello shots. They said yes! They occasionally cater events with their boozy Jello shots and cakes, which would be a good opportunity to mix in some interactive tech to wow customers.

An assortment of Solid Wiggles Jello shots. Aren't they pretty?

This semester I'm taking a class in circuit board design and production, where I'll build a standalone circuit board that will make my Jello capacitive touch sensor tech more stable, compact, and reusable. I'll be in communication with Solid Wiggles as I design this circuit board to make sure that we're on the same page about how to incorporate this tech into an event.

Jello Robot?

I tried to further push the boundaries of what I can do with Jello in my Soft Robotics class. We were making cable pull contraptions, where you control something's movement by pulling on a string - like a puppet. I tried to make a Jello octopus "robot", where several articulated Jello legs routed strings up through a head. I had some limited success with getting one leg to move.

A sad Jello octopus demo. I'm pulling on a string through the octopus's head that moves its leg

I had to place the strings inside of straws so that they had a hollow passage to move through. Then I tried to suspend the sealed-off straws in Jello, so that I could later cut them out into narrow legs. It was tricky to keep the straws fully suspended in Jello as it solidified, they kept floating to the surface because they were so lightweight. I was only able to carve out one leg to demo this concept with. I hope to return to this project in the future!

Strings inside of straws inside of Jello liquid