Multisensory perception: How sight, sound and touch shape what we taste, with Charles Spence, PhD

Why does seafood taste better at the beach? Why are so many snacks crunchy? Can the color of a room or the music in the background change the way we experience what we eat and drink? Charles Spence, PhD, head of the Crossmodal Research Laboratory at the University of Oxford, discusses the science of multisensory perception and how our senses work together to shape flavor; why food is the ultimate multisensory experience; and how insights from this research can help people eat more mindfully and enjoy eating more fully. Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome? That's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies 18 plus.

• 0:27Speaker 01

  Born from the intensity of a night in the eternal city. Introducing Born in Roma Fragrances by Valentino. Feminine perfume Donna Born in Roma features juicy black currant, central jasmine, soft cashmerin, and warm vanilla. Masculine-leaning but unisex Womo has notes of fresh violet leaf, aromatic sage, green vetiver, and mineral salt. Shop holiday gifts at Macy's.com.

• 0:57Kim Mills?

  Think about the sweetness of a strawberry or the satisfying crunch of a potato chip. What makes these things taste so good? You might think it's just sugar, salt, and fat, but research suggests that it's more than that. The color of the strawberry, the sound of the potato chip crunch, even the place where you're eating can all contribute to how you experience each bite. Psychologists have long studied the senses individually, but in the real world, sight, sound, taste, smell, and touch are constantly interacting. Today, we're going to talk to a psychologist who studies multisensory perception about how our senses work together to shape how we perceive and taste the world. So how does the brain combine information

• 1:41Kim Mills?

  from different senses to create our global perceptions? What are some of the most surprising ways that our senses interact? And how can you use this knowledge to eat healthier, enjoy more food, and otherwise improve your life?

• 1:56Kim Mills?

  Welcome to Speaking of Psychology, the flagship podcast of the American Psychological Association that examines the links between psychological science and everyday life. I'm Kim Mills.

• 2:11Kim Mills?

  My guest today is Dr. Charles Spence, a professor of experimental psychology at the University of Oxford, where he heads the Crossmodal Research Laboratory. He's published hundreds of academic papers on multisensory perception. He also brings his knowledge out of the lab and into the world. He's consulted for many multinational companies on product design, packaging and branding. He's worked with chefs to design foods that maximally stimulate the senses. And he's helped car companies design multisensory warning signals for drivers. Dr. Spence has won many awards for his work, including a 2008 Ig Nobel Prize for Research That Makes You Laugh and Then Makes You Think. He's also author of the books Gastrophysics: The New Science of Eating and Sensehacking:

• 2:55Kim Mills?

  How to Use the Power of Your Senses for Healthier, Happier Living. Dr. Spence, thank you for joining me today.

• 3:01Speaker 03

  Thank you.

• 3:02Kim Mills?

  So I just mentioned that you direct the Crossmodal Research Laboratory at Oxford, so let's start there. What does crossmodal mean in your research?

• 3:10Speaker 03

  So as a psychologist who studies the senses, sight, sound, touch, taste and smell, I spend most of my time thinking about how the senses relate one to another. And the sort of two words that we use regularly are crossmodal on the one hand and multisensory on the other. Some people, many people use those terms interchangeably, but to those who've spent their lifetime thinking about the ways that the senses can interact, multisensory is take to mean when things kind of come together into one unified gestalt or whole. So maybe the flavor of food has taste and it has smell, texture, all of those things come together into one unified flavor, which is multisensory.

• 3:51Speaker 03

  I contrast that with sort of crossmodal cases where, for example, if I suddenly slapped my hands together, that sudden loud noise will make the lights seem brighter wherever you are. You don't integrate that clapping sound with the lights, but nevertheless, the sounds crossmodally affect visual perception. So the things kind of stay separate and yet one influences the other. And over the last thirty years, I've spent a lot of time thinking about these cross-modal and multisensory effects, starting in the lab and then going out into the real world to see where they apply and then coming back into the lab again to study them some more.

• 4:25Kim Mills?

  All right, well, we'll get around to talking about that, but I want to start by noting that most of your work has focused on food and taste. Why is eating such a powerful example of multisensory perception?

• 4:37Speaker 03

  So I guess eating and drinking is something that we all do. It's a shared interest, something we all do several times a day normally. And it is probably one of the most multisensory things that we do. I mean, watching TV, that's got sort of sound and it's got sight. But smell vision's never gone very far. We can't taste the TV yet, so it's multisensory, but only with a few. Whereas food and drink are kind of inherently stimulate all of the senses. They have a certain visual appearance that might look good on Instagram. They have the sound of the crunch or the crackle or the snap or the crack or the pop to stimulate our ears, the mouth feels of oral touch, the texture of food, the spiciness,

• 5:20Speaker 03

  another sense, a pungency of pepper, the burn of chili through, what have I missed there, the smell of it and the basic tastes of sweet and sour and salty and bitter. And all of those things, not every time, but very often, all of those senses are kind of brought together to contribute to what we think of, what we describe as a taste of food. But really, we mean the flavor of food or the flave I'd like to introduce. I love the flavor of that would be much better described than I love the taste of it, because taste to the sort of the food scientist would narrowly mean just what you get on your tongue, which would be sort of sweet, sour, bitter, salty, umami, and maybe a few other things.

• 5:58Kim Mills?

  Now, you found that sound can influence taste, something that you call sonic seasoning. How does that work? And maybe you could explain your sonic chip experiments as part of the answer.

• 6:08Speaker 03

  Yes, fortunately or unfortunately, my father's no longer with us, but he would always, when he heard me talking about the sonic chip, go, not the sonic chip again. He was tired of hearing about it, but maybe not everyone has. And this was work we did back in 2004. The Ig Nobel Prize for Nutrition in two thousand and eight for a study where we brought twenty hungry undergraduates into the lab in Oxford, the Crossmodal Research Lab, and gave each of them two tubes of Pringles to munch on and put them in a soundproof box with a microphone. And each time they bit into one of those Pringles, we take the sound, pick it up from the microphone, take it outside, transform it a bit, make it louder or quieter, boost just the high frequencies or suppress them in the crunching sound and then play them back over headphones.

• 6:52Speaker 03

  Um, and we're able to show that we can enhance the crunchiness, the crispness and the freshness and hence the liking of those potato chips by about fifteen percent simply by enhancing the sound of the crunch. We never thought anything of it at the time, just published it in some obscure journal, but then it sort of got picked up by the Ig Nobel Committee as, you know, why on earth would anybody study that? Um, maybe I should get a second Ig Nobel because I'm still studying it about twenty five years later. And that, uh, it illustrates for us kind of the example of sound as a forgotten flavour sense. They're the sound of the crunch, the sound of the food itself, but really turns out there is a mystery. As the title of one article in Science in two thousand from Weiss put it, uh, why does nobody like a soggy chip?

• 7:33Speaker 03

  Or potato chip or crisp, as we call them. Um, it's got the fat, it's got the flavourings, it's got the salt. The only thing it doesn't have is the crunch, but crunch noise in food has no nutritional value. So why are so many of our snack foods crispy, crunchy, crackly, snap, crackle and pop? Um, it's sort of a mystery. Um, and from there, we've been looking both at the sounds of freshness. And maybe in fresh produce like apples and celery and carrots allowed a snap or crunch when you bite in. Signals of fresher produce which indicates, uh, greater preserved nutrient density from a fresher product. Maybe in the fried and baked category, the biscuits, the crisps, the breakfast cereals, maybe the sound of the crunch.

• 8:15Speaker 03

  Maybe it indicates fat content, I wonder, because maybe we don't have fat receptors as such in our mouths. But fat's something our brain loves for all the energy. So maybe it's a sound of crunch that we've learnt probably means it's a fatty food, energy dense. And hence we like the sound.

• 8:30Kim Mills?

  It's not just the food itself. You have found that even the plates and the cutlery we eat with can affect how we experience food. What are the factors that go into that?

• 8:41Speaker 03

  Yeah, so I guess my food research, I wasn't initially interested in studying food because that's, I was interested in the senses and I would be doing lots of warning signals work on cars and interface design and food and flavour was just what kind of what food scientists did, not what psychologists did, in my mind and many others. I got sort of dragged into food through Unilever who were funding the lab at the time and that led into the sonic chip and from there we sort of started moving out, thinking about not just what sound does food make, but also what does it look like? Changing the colour of fruit teas, say, in order to set people's expectations for flavour. If you see a very bright coloured tea, you expect it's going to have an intense flavour. A lot of the products on the market 20 years ago didn't. They looked great, they smelled great, but when you tasted them, they were

• 9:24Speaker 03

  kind of a bit thin because they didn't have enough sugar in them, as it turns out. So looking at the taste, the texture, the pungency, the sound of food, all stuff within the food itself and how the senses interacted there. And as the years have gone on, we've kind of moved out to think about the sound of the packaging, the colour of the packaging, does that affect the taste? Yes, it does, just as much as the, or maybe almost as much as the colour of the food or drink itself. And then again, to the environments in which we eat and drink, thinking about the colour, the lighting levels, the background music or soundscapes. And they're showing that all of these things, they're not part of the flavour of food or taste, and yet nevertheless, they can systematically influence the taste. So for example, we did an experiment with about 3,000 people, maybe one of the world's

• 10:07Speaker 03

  biggest wine tasting experiments in London in 2014. And over four days, we invited these 3,000 people in, it's part of a food festival, gave them all a glass of red wine and a black tasting glass, so they couldn't see it. Got them to rate the fruitiness, the freshness of the wine, how much they liked it or not, under white lighting. Then we flipped the light switch, turned the room lights to red, asked them to write the wine again. Then flipped the room lights to green, rated the same wine in a hand again, and then put the lights back to red with some sweet music, some sweet sonic seasoning in the background, and then rated the wine a fourth time. And amazingly, simply by changing the colour of the walls,

• 10:46Speaker 03

  we're able to bring out the fruitiness of the wine by putting the red lights on by about 15%. Turn the lights to green, they bring out the fresh acidic notes in the wine, and then add some sweet or sour sonic seasoning, which sort of makes no sense at all. I mean, why would room colour lighting, why would music sonically season change the taste of the food? Because it's not part of the food, and yet surprisingly, it doesn't. Now, I'm still getting to the bottom of it. I remember in 2008, we did the Sonic Chip, and around the time I got introduced to a chef called Heston Blumenthal, we put him in our booth, gave him some Pringles. He came out a changed man, saying, you know, wow, sound is the forgotten flavour sense. Sound's an ingredient the chef can use in the kitchen,

• 11:28Speaker 03

  and that led into a dish that was then served in the restaurant, in the world's best restaurant, as the signature dish, the Sound of the Sea seafood dish, that comes to the table, looking like the seashore with sand and sashimi, foam, seaweed, laid out. But the waiter brings a conch shell out of which dribbles some earbuds, and recommends that the diner put those earbuds in, whereupon the diner hears the sounds of the sea, the seagulls, the waves gently crashing on the beach. But that was sort of built on research we did together with the chef to show that these seaside sounds do make seafood taste better, but no more salty. And the sort of chef has creatively taken that insight, and then turned it into a dish that has since been sort of replicated around the world, and hopefully gets diners coming out of the restaurant if they're lucky enough to go, thinking for the first time, yeah, sound is important.

• 12:12Speaker 03

  And I never thought about that before, and has this amazing effect that for some number of journalists who write about going to the restaurant, and even for people who are lucky enough to get their 50th birthday party for a once-in-a-lifetime meal, they come away and they say, I cried.

• 12:27Kim Mills?

  You worked with a chef who had this meal, I think he was in a restaurant in Milan, and we're talking about making people cry. I mean, when you got to the, it was a multi-course meal, right? When you got to the sixth item, which was rice pudding, how can rice pudding make people cry? I mean, what was it about that, that particular dish, the way that it was all put together? Have you looked at it?

• 12:49Speaker 03

  Yeah, so kind of going back to this sort of Sound of the Sea dish that came onto the menu in the World's Best Restaurant in 2007, we had no idea it would make anybody cry. But as the newspaper reports came in about this dish, and we saw it was happening, I've been sort of collecting examples of these sort of extraordinary responses to food, dishes that make people cry, or that are awe inspiring, or magical, or give you an ASMR tingle, one of those neck shivers you get when people whisper, "Minecraft paper." And just by chance, I got introduced to Federico, the chef in Milan, his restaurant Sensorium. He's a young chef. Every year, he comes out with a new menu, sort of 10 courses thereabouts, and you all sit at the counter in front of the chef. And for this ayahuasca themed menu, kind of all the, those sort of mystical drugs of the, of the Amazon

• 13:33Speaker 03

  without the drugs, was his concept. And of course, number six, as you say, when he started serving this ayahuasca themed menu, people started crying. And it was not what he thought intended either, it just sort of happened, and they never cried for any of his previous menus. They don't cry on the first course, or any other courses on this menu, it was only course number six, the rice pudding dish. And when I got to hear about that, I was lucky to get to eat at his restaurant, experience the thing for myself. I didn't cry, have to say. And then we got talking to the chef, and I've been able to do experiments in his restaurant with his diners on that dish to figure out what's going on. I think nobody cries in a restaurant when you first walk in the door. You need to relax and feel comfortable in the space. So it's probably that this course six out of 10 is important.

• 14:17Speaker 03

  You're relaxed by this point. And the chef kind of creates an emotional journey, because it's like two and a half hours, three hour meal. So a bit like, I know, a movie, and you have all these emotional trajectories of storytelling, and his, this menu is on the kind of Cinderella thing he describes. Things get good, and then they drop down to really bad and chaotic. And it's just after that really bad moment that- When this rice pudding occurs, so on this emotional trajectory, once you're relaxed, that's when it happens. And the dish comes to the table as rice pudding, the waiter waitress will break like a wafer over the top of the bowl for you. So I wonder whether that's kind of that Eucharist sort of religious rituals going on there, maybe. And as well, there are sounds, there's some music and there's like nostalgic sounds, there's an old man recounting his life from one loudspeaker.

• 15:03Speaker 03

  There's a woman on the other corner, like a school mistress with little children running around in a school bell. And then at the same time, there are also some very low frequency vibrations like sub audible naught to 20 hertz. Sound waves, but you can't hear them, but they make your body feel something like kind of tingle almost. You're aware of your body for more than normally the case. And so with the chef, we've been giving a hundred people the full experience, asking them whether they cried or felt like crying. Did those who cried rate the food as tasting better? It's kind of important, which they did. And then another hundred diners without those low frequency vibrations that make you aware of your body. And then a hundred delivery drivers who rock up with this week's tomatoes. So would you try some rice pudding? Yes. Did you feel like crying?

• 15:48Speaker 03

  What are you talking about? I have no idea. Don't be stupid. So none of the delivery drivers cried. Suggesting it's not just about the rice pudding itself, it's something about the whole kind of gestalt of the experience. And that when we turn those bodily vibrations off the low ultra low frequency vibrations. Then the rate of crying drops significantly, suggesting that's kind of a part. So we think it's probably. These vibrations make you aware of your body and when you're aware of your body sensations. And you're in this sort of nostalgic emotional frame, then it all kind of comes together. And yeah, just the tears start flowing for you know, two thirds of diners.

• 16:26Kim Mills?

  That's amazing, but that's the kind of thing where you can only do it in the real world, right? I mean, you can't really recreate something like that in a laboratory.

• 16:33Speaker 03

  Absolutely. And it's kind of one of the funny things that these days as psychologists, we always have to fill out endless ethics forms. Before we undertake research.

• 16:44Speaker 03

  And you know, if my ethics panel got to hear that my subjects cried, my participants, sorry, these days cried, I'd be in big trouble. I'd be hauled up, what are you doing? How dare you do that? It's not ethically justified. Whereas the creatives, I think I know artists, but also the chefs, mixologists, they're both perhaps more capable of eliciting these extreme, strong emotions and then dealing with the consequences. They have the tissues on hand. And so, yeah, this is the sort of thing that works much better there. And they probably couldn't recreate in the lab. And I sort of remember when we were doing the sound of the sea experiments, we, the chef came up to Oxford, to the science museum here, and we had 150 people for our conference. They'd pay some money, they wanted some food.

• 17:30Speaker 03

  He brought up a van of oysters. We did the sound of the sea versus the sound of farmyard chickens, showed that the sea sounds things better. And then he also brought up some bacon and egg ice cream, which was a dish in the restaurant at the time. Very strange bacon and egg and an ice cream, it's kind of all kinds of wrong, but sort of molecular and modernist. Gave people that to eat while they heard the sounds of sizzling bacon or the farmyard chickens, we could flip the flavour. And then that's sort of great, the dish works very well in the restaurant. But then we had some more batches of the bacon and egg ice cream that I would give to participants in the lab for some follow-up, carefully controlled studies. And there, people go, oh, that's horrible. In the wrong context, it's like, that's just really nasty ice cream. If you know it's like a designed experience from the creative chef or in a design shop,

• 18:15Speaker 03

  then you're much happier to, you may even expect to be surprised or confronted by sensory incongruency. So there it works, but only there can you study it. And things like, you know, surprise, when you bite into something that looks like an apple, that tastes like a pineapple, or that you can only do once. And again, that's very hard to capture in a lab situation when somebody comes in a white coat and gets you to fill in the consent form and is monitoring you and the whole atmosphere is not conducive to maybe that more, how should we put it, emotional or engaged response or something.

• 18:50Kim Mills?

  Yeah, a few years back, we, on Speaking of Psychology, did an episode on synesthesia, in which the researcher, Dr Julia Simner, talked about how people with synesthesia can taste certain words, which can even taste nasty. She said she had one subject who thought her name was nasty, and so he wouldn't even say it. So how is what you study different from synesthesia?

• 19:14Speaker 03

  So there are all these things like the crunch of the crisp and it's all kind of part of the thing itself. That's sort of maybe multisensory. And then there are these more surprising connections between the senses, such as, you know, when I ask you to think of what's the sweetest color? Most people will say pinkish red or something. If I ask you to say, you know, is sweetness round or angular? People think it's going to be sweet, it's round. Bitterness is angular. And these are like these surprising connections between the senses. And that's a bit like synesthesia, but it's, I think they're fundamentally different and dual. So I was an undergraduate with here in Oxford back in the eighties. Yeah, we have debates about this.

• 19:56Speaker 03

  I think synesthesia for a long time played a really important role in stimulating artists, creatives, Nabokov, Kandinsky, Skriabin. You name it, but the synesthetic connections are kind of unusual. They're idiosyncratic, you know, one synesthetic experience is blue for Monday, the next synesthetic says, no, it's orangey, gold. And so the connections are different, whereas what I'm studying is this kind of universal synesthesia, which I call cross modal correspondences. These surprising connections between the senses that we share between tastes and colors and shapes and textures and sounds and timbres and tempos and rhythms and roughness. Very surprising, but really interesting.

• 20:36Speaker 03

  And so for in our research, when we, after we did the sound of the sea experiment that led into the dish, that was, we'd never have done that study. If there hadn't been 150 people turning up the next day and they demanded some food, we didn't think of what to do. It would be too crazy to think that playing seaside sounds would change the taste of, after a few years of the dish being on the menu, people started saying, well, it's obvious, isn't it? You know, seaside sounds make, you know, fish and chips taste better by the seaside. We didn't need your psychologist to tell us that.

• 21:01Kim Mills?

  Right.

• 21:02Speaker 03

  So we had to try and think of the next new thing. And that became this sort of sonic seasoning narrowly defined, which is when we use music. Uh, to change taste of food. Because if we know that uh, sweet tastes are more high pitched, more like to be piano sounds, uh, bitter tastes, that's for the lower the pitch of the music or sound, the more bitter, brassy sounds are bitter. And now we have these whole musical menus for bitter, spicy, creamy, sweet, salty, umami of exactly what the music or sound scape should have in terms of tempo, rhythm and so on. And then we can either find music off the shelf that has those qualities or get a composer or sound designer to create composition to match, accentuate the taste.

• 21:46Speaker 03

  And this is yeah, often described as synesthesia. Uh, certainly back in the 1930s, the Italian futurists, people like F T Marinetti, he's got a whole futurist cookbook, where you find most of modernist cuisine, uh, in an idea, if not an execution, and probably a horrible man, sort of misogynist, uh, and all sorts of bad things. But in the futurist cookbook, there are all sorts of great ideas and he talks about syntactilismo, synauditivo, synesthetic sound, synesthetic touch. They thought it was synesthesia then. I think they were wrong about that, but they were right that textures and tastes correspond. We can study them in the lab and then work with chocolatiers, wine brands, coffee designers, plateware manufacturers, such that we can, yeah, by changing the texture on your coffee

• 22:30Speaker 03

  cup to be rougher or smoother, we can then change your experience of the body of the coffee, which could be a good thing or a bad thing, depending on the brew you're drinking and what kind of taste world you live in.

• 22:41Kim Mills?

  So do you do dinner parties where you curate the music to go along with what you're serving?

• 22:46Speaker 03

  Absolutely, of course, uh, certainly we do. Um, I have to contend with Mrs Spence, um, so a lot of my ideas about gastrophysics, uh, I'm not allowed. All my black angular plates, I've got a very good scientific reason why they work for certain foods, but she doesn't like them, so they're out on the street. A few things stick though, but once a month she will let us have like a multisensory lab dinner party and then we all get to have some fun. We've had some chefs in residence over the years and they would cook away. I remember one memorable occasion where we had Rupert Ponsonby who does beer marketing and he brought a whole range of beers and we had my wife's ironing table, all the beers arranged on top and a tone generator and

• 23:28Speaker 03

  we sort of recreated Holt Hansen's 1969 Pitch of Harmony experiment when he brought people into Copenhagen University, gave them Carlsberg regular lager and Carlsberg elephant export lager that's stronger and said just twiddle with the tone on the tone generator till you find the frequency that matches the taste. People think it's bizarre, but then with 10 people and sort of 10 bottles of beer, you find that people match the stronger, more alcoholic export lager to a higher pitched sound. And so we sort of recreated that. And again, he in 1960s and 70s was showing that when you played the pitch of harmony to people as they tasted the matching beer, then they had these sort of experiences

• 24:10Speaker 03

  where their hands were shaking, their heads were expanding, the kind of research that I guess made psychologists not ask their participants what they thought anymore. Just fill in the question, just give me a 1 to 10 answer. That's all I want to hear from you. Whereas in the old days, you'd ask people, you know, how was it for you as a subject? What went on? And you got all these weird responses. And ever since I first came across that study in the library when I was an undergraduate, I wanted to reproduce it. And we had it there in the lab dinner while the chef was was cutting potatoes into 3D sculptures, just like Marinetti was doing in the 1930s. And it's turning my wife's Kristoff, like very expensive silver cutlery. And he'd been to the market, the chef, and he'd bought, I'm not sure, a bunch of hairs.

• 24:53Speaker 03

  With their pelts and they wrap the skin and the hair of the pelt of the hairs around my wife's cutlery and then we had served a hair stew and anybody who's there will still remember it that you're sort of feeling the animal, the warmth of the fur in your hand of the spoon, you could smell the animal and you're eating it and it sort of makes you much more mindful about what you're eating and it is kind of a syntacticism, it's kind of touch and taste coming together, just like in those futurist dinner parties. And yeah, lots of fun was had by all and amazing how many, that's just like a chance to play really and to see what works and some of the dishes that have been served in those multisensory lab dinners then end up being kind of formal research that we follow up

• 25:36Speaker 03

  in the lab with larger sample sizes who are blinded to the purpose of the research and so on and so forth.

• 25:42Kim Mills?

  We're going to take a short break. When we return, I'll talk with Dr. Spence about how distracted eating changes our experience of food and about what makes food trends go viral.

• 25:56Speaker 00

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies 18 plus.

• 26:23Speaker 01

  Born from the intensity of a night in the Eternal City. Introducing Born in Roma Fragrances by Valentino. Feminine perfume Donna Born in Roma features juicy black currant, central jasmine, soft cashmerin and warm vanilla. Masculine leaning but unisex womo has notes of fresh violet leaf, aromatic sage, green vetiver and mineral salt. Shop holiday gifts at macys.com.

• 26:54Kim Mills?

  I want to ask you, a lot of us, particularly in the States, maybe you do this in England as well, a lot of us eat in front of the TV. What is the impact of eating in front of the TV on how you experience your food?

• 27:07Speaker 03

  Mostly bad, bad, bad. From the Gastrophysics book, that was one of the strongest relations that came out. It was from research showing that we eat significantly more when distracted by technology. Quite how much more? It could be as much as 30% more when eating with the TV on than off. It all depends whether it's a new show or a repeat and how entertaining the program is as to how much more, but yeah, significantly more with a TV on. And these days, I guess, a lot of people will eat with their mobile device instead. And there, something like 10 to 15% increase in consumption. So I think generally very bad, in part because those technologies just distract us from what we're eating. And I think our brain is going to use the sensations associated with consumption to

• 27:52Speaker 03

  figure out when we should stop eating. So if we're, if the sense, all the sensations we get are from the TV or on mobile device, we're not aware of the taste and the textures, then it takes more food before our brain says, okay, I've had enough food sensations, it's time to stop now. And that's kind of why you end up eating more. That's definitely the bad side, but I do think there's sort of potential here to turn it for a force for good sometimes. So we did a project back in 2017 with one of the home food delivery companies there in the UK. And I'm not sure whether it was Netflix or one of the other streaming services. We knew all, it was Game of Thrones, it was, there was like a mafia TV series at the time and some other shows.

• 28:36Speaker 03

  And we knew what was going to be airing the next episode. So worked with a chef to make the meals that matched what the characters on the shows you might be watching on the streaming service would be eating. And there, I think there could be a possible sort of resonance that you are almost, you know, by eating the same thing as the characters on screen are eating, that sort of connects you and maybe rather than distracting you, turns it into a more immersive, engaging, multisensory experience than otherwise would have been the case.

• 29:03Kim Mills?

  So you were recently quoted in an article talking about Dubai chocolate, which is everywhere right now. What is it about Dubai chocolate that in a multi-sensory sense has made it so trendy?

• 29:18Speaker 03

  I think it's all about the eye appeal, the Instagrammability, not really anything to do with the taste or the texture or the chemicals. You might like it or not. For me, it's too sweet. It's kind of this sort of a chocolate on the outside.

• 29:32Speaker 03

  Maybe wrapped as if it's a gold bar, a bling appeal. And on the inside, it'll have this very bright green kind of pistachio and other stuff, kind of crunchy interior. Didn't exist, you know, sort of five years ago and sort of exploded. So that wherever you go now, like in a European capital, you'll see that green and brown imagery for smoothies, for cakes, for hot drinks, for ice creams, for chocolate, for you name it. So how, what's going on? I think it's really all about Instagrammability in that if I discovered a new regular chocolate bar that was the best tasting chocolate I'd ever had, I couldn't share that with you over Instagram or social media. The picture of it would just look brown, just like any other chocolate. So it's impossible to kind of communicate visually, digitally deliciousness.

• 30:18Speaker 03

  Whereas the Dubai chocolate has this distinctively different eye appeal, the green and brown. That color combination doesn't really exist in anything else. And hence, when I see it, I can uniquely index that food. And hence, it's much more Instagrammable. And I think successful. Paired with, you know, other things like Aperol spritz has had a moment in recent years. And Provencal rosé wine has had a resurgence of popularity in Europe. And again, for both of those drinks, the idea is they look good on Instagram or they look distinctively different. If you saw a clear drink, it could be gin and tonic, it could be Sprite, it could be soda water, you wouldn't know what it was. But that kind of orangey of Aperol on a summer's

• 31:02Speaker 03

  evening, somewhere in a piazza in the Mediterranean, then it can only be one thing. And it's that distinctive difference that helps these products to really become viral. And probably also explains, then, if you have a product that doesn't look distinctly different, the question of why things like extreme spicy challenges, extreme sour challenges are becoming a thing. And there, I think it's why those two tastes. Well, because those are things that I can express through my facial pain, eating a very hot chili food, you will see me sweating and a very sour food will make my face pucker up. And so in a way, I can communicate those tastes through the extreme response in my face in a way that I couldn't with sweetness. So it's going to look different,

• 31:42Speaker 03

  or evoke in you, the consumer, some extreme physical response that can then be conveyed through media. And that these things will sort of, I think are becoming more and more rapidly emerging as sort of AI health starts to pick up much more rapidly on emerging food trends around the world. Matcha tea, or bubble tea, or the Dubai chocolate, picking up on the trends more rapidly and allowing food companies to deliver to market their new version of Dubai Chocolate X in time to kind of meet the ascending wave of popularity. Whereas in the past, by the time the food company had heard about this new trend, it's kind of too late already to ride the wave.

• 32:27Kim Mills?

  Speaking of AI, you have any thoughts on how and whether AI is going to be, say, designing new dishes or entire cuisines?

• 32:36Speaker 03

  Yes and no. I think not in a way that so far people have thought about it, which is there's been a lot of excitement about the idea of sort of flavour pairing, the idea that you could combine ingredients effectively in a dish, if you know what molecules and volatile compounds, chemicals they have in them, and two foods that share similar chemicals will pair well together. So you get things like caviar and white chocolate share a molecule, indole and jasmine and pork share a molecule, so they go very well together, cauliflower and dark chocolate.

• 33:12Speaker 03

  Or maybe it's blue cheese, they go together. So you get some very unusual combinations that work well, and the idea being they share a molecule. And with AI and large language models, that seems to make the whole process of pairing flavours based on shared compounds much easier than looking up numbers in charts and things. And yet, I think that's not going to work in the end, because I don't think flavour pairing is generally a sound principle. And certainly when people have studied it systematically, foods are not likely to go well together just because they share a compound. But instead, I think the way that AI is already starting to change the food landscape is with, illustrated by a chef we work with in London

• 33:52Speaker 03

  called Joseph Huser. And he very often gets invited to make like new foods for Virgin Galactic over in Arizona, when they had the party for the people who put a deposit down being on the space first space tourists. Or when the Willy Wonka movie came to London for its launch with Timothy Chalamet, whatever, a year ago. He was asked in both cases, can you make some canapes to match the event? Very often, when he makes those things, then the client says, okay, that's too adventurous. Just give me the cheese and pineapple sticks. And he's wasted all that effort. So what he started doing, he's really clever, I think, is asking AI to generate the visual concept of a dish. So he types in for that, say for

• 34:33Speaker 03

  the Wonka movie, he says, give me a chocolate hairbrush. And then AI comes up with this amazing sort of chocolate, chocolate eclat with hair. If the client likes the idea from just on the visual image, then he's got sign off. And then he goes and figures out how to make it and sources from around the world ingredients. So to get the hair on the chocolate de clair hairbrush, that was going to Turkey for some Turkish bizani or something. They have some sort of food stuff that matches. And that's been very successful for him as a, as a kind of aid to culinary creativity, rather than a replacement of that, yeah, maybe it's, it's really good at sort of generating ideas. But any of those ideas that are generated

• 35:16Speaker 03

  by AI, given that AI doesn't taste nor smell, have to be then sense checked by the real creative of the kitchen in order to make sure that they work or otherwise. When we did a study last 2024, we got lots of Instagram images of foods, raw foods processed and ultra processed like carrots, carrot juice and carrot cake. And then we made a AI version of the same foods and asked whether people could tell the difference. And in 2024, with the software we used, people could say, yep, that's AI-generated food image. As such, we don't like it as much as the real thing. But roll the clocks forward now, and with every evolution of the AI, it gets harder and harder to tell the difference,

• 35:58Speaker 03

  and more and more tempting for the food companies in the marketing to present AI-generated dishes. And the real concern for me is that maybe they will be generated to be ultimately appealing to our brain, because, you know, the AI will to realise that if it's the image of the food should be from a first-person perspective, and look large, and have yolk porn, with their protein in motion dribbling out, and it should have lots of energy density, because that's what our brains love. And it will create these hyper delectable food images, that even if we do not eat, the images that we see, they may nevertheless set the standards for what we think is acceptable in a meal. And that's going to go all the way through to, one of my favourite examples from

• 36:39Speaker 03

  the last year or two was a restaurant in Austin, Texas, Ethos, I think it was called, the restaurant. And you go online, it's got these amazing images of like a croissants in the shape of potmuses, that look huge, and then sort of flower arrangements of sushi, amazing. And the restaurant manager who can take your booking for a table, called, was it Giuseppe Fusilli? And the whole thing's AI-generated. There is no restaurant, there is no restaurant manager, he's a bot. And all these food images are, but they're sort of so appealing.

• 37:11Kim Mills?

  Everybody wants to eat there.

• 37:13Speaker 03

  Yeah, everyone wants to eat there. I know we've all been tricked by one of these sort of digital hallucinations. And yeah, it's going to become harder and harder to watch out for them.

• 37:23Kim Mills?

  So just to wrap up, what's next for you? What are you studying now? What are your big questions ahead that you want to answer?

• 37:29Speaker 03

  Oh, all sorts, all sorts of stuff. Maybe one of the big ones is, when we've tried over the last decade or two to create these multisensory experiences by adding an extra sense, some music or lighting to something, or touch, but it's often been just trying to add another sense to kind of match with what's already there. So we pick a sound that matches the taste, that's congruent with it. But I've recently been looking a lot at the literature on sort of media studies and film music. Design, an area I haven't come across before, but it's its own separate world. And there to see all the clever ways that sort of movie directors and those who create the compositions for movie scores will use sound sometimes in sort of a counterpoint.

• 38:13Speaker 03

  Sometimes they'll have, you know, really violent scenes on screen. Quentin Tarantino and Stanley Kubrick, sort of famous examples, where when they have ultra violence on screen, they might have, you know, um, singing in the rain or something, happy music as a sort of counterpoint. So could you have a culinary counterpoint? I don't think anyone's tried that yet, having music that deliberately goes against something in the dish and makes sense only as a kind of composite. And then these other examples of Anna, Anna, all these big words I have to look up and I forget now, Anna Mathia or something, where the music starts and then something happens on screen and then the music keeps continuing. It's like the musics doesn't care about the world, doesn't care about what's going on. It continues regardless of these terrible things.

• 38:58Speaker 03

  And again, it is, could you use that sort of notion in a dining concept and what would it mean? I think there are lots of these ideas from this kind of mood music in film to think, could we translate that into a multi-sensory dining experience to create new kinds of things where we don't just match what's already there, but do something different, something that augments it. And then maybe ultimately we can sort of deliver this kind of idea of a multi-sensory gestalt or the total work of art, being very ambitious, that does sort of engage all of the senses and only makes sense when all the senses kind of come together. We'll see if we ever get there, but a worthy challenge that'll keep me going for a good few years, yes, I should say.

• 39:40Kim Mills?

  Well, I want to thank you, Dr. Spence, for joining me today. This has been just amazing and now I'm hungry and I have to go find something to eat.

• 39:47Speaker 03

  Thank you.

• 39:49Kim Mills?

  You can find previous episodes of Speaking of Psychology on our website at speakingofpsychology.org or on Apple, Spotify, YouTube, or wherever you get your podcasts. And if you like what you've heard, please subscribe and leave a review. If you have comments or ideas for future episodes, you can email us at [email protected]. Speaking of Psychology is produced by Lee Weinerman. Thank you for listening to the American Psychological Association. I'm Kim Mills.

• 40:55Speaker 04

  This episode is brought to you by State Farm. Football has a play for just about everything. You can go deep, keep it on the ground, or draw up something nobody sees coming. And whether you want to run a flea flicker or the tush push, you need a plan to fit your unique situation. That's why State Farm takes the time to get to know you, so they can help personalize coverage for your life and budget. Now that's a solid game plan. Like a good neighbor, State Farm is there.