“Smell can be art, and it also can be science”: AI/ML and digital olfaction

Danny Crichton:
Hello and welcome to Securities, a podcast and newsletter devoted to science, technology, finance, and the human condition. I'm your host, Danny Crichton. Today, we're going to talk about smell. And I know that's a little bit ironic on an audio podcast, but smell is so intrinsic to the human experience. When it comes to our senses, we can obviously watch movies, we can obviously listen to bands and each other, but when it comes to triggering our memory, the nostalgia for the past of places we've been and people we've met, smell is where it's at. It is the thing that triggers all that emotion in us. And, yet, we actually know very little about why smell works or even how to represent it mathematically.

We've actually known for more than a century how our eyes work, how to represent color in terms of mathematics. And ditto for audio, we know how our ears function, we know how they work. We have an audio representation of audio wave forms. But when it comes to smell, we have none of the above. We really don't understand the neuroscience of it, and we have no mathematical basis by which to capture smell and transmit it to other places. And that's a real shame, because when it comes to building unique human experiences, smell is so important.

I remember a couple years ago at Disneyland, at the California Adventure ride, you actually are sort of flying around California, and at one point you're going through the Central Valley and you fly through an orange orchard. And as you're flying, the Disney Imagineers have sort of creatively spray you with this orange smell that really triggers this notion of you're here, you're having this unique experience, and there's really nothing like that out there.

And so you have this magic of smell, and yet our computers remain completely oblivious to it. How do we create digital olfaction? How can we build a bridge from some of the frontier neuroscience that's going on in this field to actually help our computers have a sense of smell? And about a year ago, we ran into an entrepreneur, a SaaS research scientist at Google Brain named Alex Wiltschko, who has had his life's work focused on this subject. From collecting perfume bottles in high school, all the way through his neuroscience PhD, Alex has been focused on figuring out the neuroscience of smell and connecting that into our digital devices.

And so we funded him. And about a year ago, he became the CEO and founder of Osmo, which is focused on bringing digital olfaction to computers. And today we're going to talk about the neuroscience of smell. Alex, thanks so much for joining us.

Alex Wiltschko:
Thank you for having me. It's a pleasure to be here.

Danny Crichton:
I'd be curious, just to get started here, how did you get into this whole category of the science of smell?

Alex Wiltschko:
It goes back quite a while, and I also went to that Disneyland experience. And I also remember the blast of the oranges. It makes it so much more immersive to have all of your senses engaged. It just really pulls you into the moment. So my I guess you could call it obsession with smell goes back before I have memory. Smell was always one of the most powerful senses for me. So when I close my eyes, I actually have a pretty difficult time imagining images. I can't really make mental imagery very well, but I've always been able to remember smells, tastes, sensory experiences like that.

And I think everybody has a different makeup as to what senses make their way into their mind and into their consciousness. And for me, smell has always been very, very present, very powerful. I started collecting perfumes when I was in high school. I was already a computer nerd. I was growing up in a small Texas town. And so those things combined to make me not a very popular person in high school. But nonetheless, my obsessions persisted, just because it really had a hold of me.

And I actually remember the first perfume that I bought that made me realize that fragrance was an art, and it was Bvlgari Black, and it looks like a little hockey puck. And it was recommended by this book called Perfumes the Guide, by a really prolific and talented perfume critic named Luca Turin, who also happens to be a scent scientist. The way that he described the scent made it sound like there were multiple acts, like a play by Shakespeare. And then when I finally got the perfume, I had saved up enough to get it off of Amazon. They had just started selling fragrances. It indeed had those acts.

It started out with a rubber tire kind of harsh, like a screeching kind of race car kind of a smell. And then it moved into this leathery kind of tobacco vanilla-y smell. And then the leather armchair on fire smell kind of faded, and then the vanilla kind of took hold, and it really did proceed, and the whole perfume was over in 45 minutes. So it's not actually a very good fragrance, because it doesn't last that long. But it was the first time that I experienced that this is not random. Smell can be art, and it also can be science. And I ended up really kind of stuck on this for my entire academic training. So I did an undergraduate degree in neuroscience, and then I did a PhD in neuroscience. And all through that it was focused on smell.

And what really frustrated me is how little we knew. We knew so much about vision, about the visual world and about how our brains process vision, same with hearing. And then for smell, it was just crickets. It was almost nothing. And there's actually this book, it's the canonical textbook for neuroscience called Principles of Neuroscience by Kandel. And it's like four inches thick.

And I took a caliper and I actually measured the thickness of the book that was dedicated to each sense. And it's like 0.75, three quarters of an inch for vision. It's like a little bit less than that, a half an inch, for hearing. And then touch gets maybe a quarter inch or less, and then smell gets 30 pages. And most of it's wrong. It's like a millimeter of information in this four inch thick tome, and it's out of date. And we just don't understand how our brain processes olfaction.

We didn't understand how olfaction was organized in the world. We don't have a map for smell like we do for vision, like RGB is a map of color. So all of this has just been swirling around in my head for years. I mean, I've been thinking about this stuff for like 15, 16 years. And little did I know that at the same time, Josh was also thinking about this from his perspective, living his life. And then we finally converged less than a year ago, and realized that we're both completely obsessed with this topic. It was a really natural match, and we founded Osmo.

Danny Crichton:
So you use caliper to measure your textbook. So I can see why you were unpopular in a small town in Texas.

Alex Wiltschko:
Oh, you think.

Danny Crichton:
That's-

Alex Wiltschko:
What else gives it a way?

Danny Crichton:
But I'm curious why are neuroscientists giving such short thrift to this particular sense? I mean, I sort of understand vision is obviously really important from a medical condition. People who have vision problems, want them solved really aggressively. Smell is so intrinsic to our taste and our memories. We've known that, I think, for a long time. Why has there not been the same effort around smell that there has been around other senses?

Alex Wiltschko:
It's a really good question and I don't think anybody's going to give you the definitive answer, but I notice a couple threads. The first thread is we've figured out how to describe at least the color part of visual world with really great structure, really great predictive power. And that's over 200 years old. So Goethe came up with color wheel, principles of color theory, and we've really developed those. And since like 1929-ish, when we had the CIE XYZ and RGB color spaces, we had a really strong mathematical background and explanation for color.

And since Fourier came up with a Fourier Transform, we had a really natural space for thinking about sound. These are one dimensional and three-dimensional spaces. Easy for us to put on a piece of paper, easy for us to think about, to visualize, smells not that way. Smell is at least two orders of magnitude higher dimensional. We've got three different channels of color information in our eyes. We have at least 300 different channels of olfactory information in our nose.

So the hope of putting a map on a flat piece of paper, using the tools of 19th or even 20th century math or computer science, just not going to work. It's just not going to work. So I think that's one piece is in order to understand how the brain is organizing the outside world, you have to have a model of the outside world, in order to try to explain or analyze the data that you collect, when you put wires in the brain and you hear neurons popping.

The other thing is there's this myth, this pervasive myth that we're not very good at smelling things. That's wrong. That's just absolutely wrong. We were really good at smelling things. We can identify the equivalent of a teaspoon of material in a whole cathedral. I mean the amount of sensitivity that our noses have exceed the analytical instruments that are in the labs meant to analyze smell. You have to go to these extreme, extremely huge, extremely expensive instruments before you even start to match human olfaction. It's incredible what this thing between our eyes can do.

And so I think that the instruments never really surpassed our olfactory capabilities. And I also think that this myth that we're not very good at it has confused the whole situation. That somehow scent is a lower sense than vision or hearing. And in fact, it's the opposite. Sense is older way, way older. I would say arguably it's the oldest sense on planet Earth. I mean when we started learning how to eat food as little bacteria, the first thing we had to do is be able to sense the food we were going to eat at a distance, which is what smell is, and be able to sense predators or potential mates.

I mean, chemical sensing at a distance is fundamental to life on earth. And in fact, if you look inside of the brain, smells organized differently than the other senses. Every other sense goes through this central way station called the thalamus. It takes extra time to become conscious of the information in the world that comes through those senses. But for smell, it goes right into the olfactory bulb, and then right into your memory centers. I mean there's almost no delay between when you sense a scent and when you are able to evoke memories or emotions that are tied to that scent, which is why it's such a powerful, powerful way to experience the world.

Danny Crichton:
Well, I hadn't thought about that. I know, I think our eyesight is what, 50 milliseconds or 200 milliseconds of delay, as it's going through the visual process and cortex and kind of coming back out. But I hadn't thought about the fact that it's basically a shortcut.

Alex Wiltschko:
Oh, yeah.

Danny Crichton:
It's not going through all the circuitry, just it's a raw feed right into our brain. And then you know, talked about the fact that it's sort of a lower scent, but then we're always told that dogs supposedly smell orders magnitude better than humans, and we sort of have this degraded sense of smell. And so I'm actually kind of a little bit surprised, because I don't actually know this space that well that we actually do. Because I always put it in this context of animals are so much better, we kind of weakened our sense of smell over time. What you're saying is we're still at the peak, in terms of technology, we're still competitive with the sensors and instruments that are available in the lab.

Alex Wiltschko:
Absolutely. And there's three differences between humans and dogs. Two, I think put dogs above us, and one we're actually pretty similar. But dogs have this long nose and that's really helpful for separating molecules out. So just having more surface area for chemical sensation and having it drawn-

Danny Crichton:
Pinocchio, would have a great sense of smell.

Alex Wiltschko:
Amazing.

Danny Crichton:
The more you lie, the more he would be able to-

Alex Wiltschko:
The more he lied, the better he smells. That's definitely how it works. So there's that, there's the actual instrument that they've got. And then they also have more channels of olfactory sensing information. They have more olfactory receptors, different types of them that are expressed in their nose. So we've got 350-ish and dogs have over a 1,000. It's not that we don't have those genes, we actually do have a lot of those genes, but they're nonfunctional. So in our genomes, the extra receptors are kind of like Roman ruins. They're not being cared for. And so they've broken down. They've become what's called pseudogenized.

So we have the potential, although those genes are broken. But dogs are actively using more olfactory receptors. We don't know what that means exactly. It probably means they can smell more things in the same way that bees can see into the ultraviolet, and snakes can see into the infrared. Perhaps, dogs can perceive things that are invisible to us, both in terms of the concentration, but also just types of molecules, that even if we were being slammed in the face with it, maybe we wouldn't sense it. We don't know. We just don't know.
The other thing is dogs are closer to the ground and the really good smells come up from the ground. All the plants grow from the ground, dirt's on the ground, people's footsteps litter everything.

Danny Crichton:
I was going to say, I live in New York City, I'm not sure I want to be any closer to the ground than I need to be.

Alex Wiltschko:
Exactly, right. So because there's so much stuff that's emanating, you can be as tall as you like and still experience the full-frontal effect of walking down Broadway in New York City. But for most of the world, these scents, they stay close to where they're from, which is planet Earth.

Danny Crichton:
And then before we jump to the modern world and all the stuff that you're doing, I'm curious, so you mentioned 350, basically, sensors in the nose. When do we have this model, this more simplistic model of understanding kind of the basic anatomy of the nose, the nasal folds, the olfactory bulb? When did that come together in this kind of simple intellectual model for how smell works in our brains?

Alex Wiltschko:
I think we knew the anatomy for the nose for some time. And the notion that information from the nose passes into the bulb and then goes into the brain is as old as a lot of anatomy itself. The detailed dissection of what different layers in the smell cortex, called the piriform cortex, do, frankly, that's still emerging. I think that our understanding of what different stages of olfactory processing do that's still happening.

I mean when I was in graduate school, just slightly after I graduated, I think one of the better papers explaining exactly what's happening in piriform cortex came out of the lab that I studied in, Professor Bob Datta. It's a paper by a former lab mate of mine named Stan Pashkovski. For the first time they actually saw what these different layers of olfactory cortex were doing. And maybe this is another reason why it's understudied as a field of neurosciences, it's really hard to get to that part of the brain.

So in order to get to it in mice, you can't just poke a little hole on the top of the skull and take a look, like you can with visual cortex or auditory cortex. You kind of have to go from the bottom. And there's a lot of stuff in the way and that's really, really difficult surgically to get access to it. So very few people in the world have the skill to actually access the part of the brain that is coding for smell. So it's this weird anatomical quirk that by being so [inaudible 00:14:32] front and protected, frankly, in the brain, it's made it harder to study that.

Danny Crichton:
That is so fascinating. I actually worked in an autopsy clinic for a little while, and I can actually see that. It's so fascinating to me to think that we think of science as we have these plans and we should focus on what matters most, and think of it as this national science strategy of what you do. And then in reality it's like, "Well, this was an easier side of the brain to actually pick out in the lab mouse and whatever the case may be. And therefore, we studied that first, because that was just what was easiest to acquire. And grad students are grad students.

Alex Wiltschko:
Total path dependence. Total path dependence. You've got it exactly right.

Danny Crichton:
That's amazing. So let's jump ahead into the modern world. So we have this challenge. So in the audio world, it's sort of linear. In the visual world, we have sort of the Roy G Biv, RGB, of the visual cortex in sort of a three-dimensional way. And then as we get into smell, we're getting into this massively orders of magnitude, multidimensional kind of problem space.

And so as you said, we can't use traditional technology. You can't use a notebook, you can't kind of track things, you can't just put on a wall and project it and kind of look at a graph. You had to come up with a new map and layer to understand the space. How did you go about finding and charting the territory here in smell?

Alex Wiltschko:
What we originally set out to do, and this work started while I was a group leader in Google Brain, and we had this really big visual understanding group, this really big auditory understanding group, I thought it was time to start a smell understanding group. So I started the digital olfaction group. And the core problem, the thing that was in the way, and this is a century year old problem, the thing that was in the way of understanding why things smell the way that they do is what's called the structure odor relation problem.

If you can draw a molecule as a structure, the ball and stick model, where the atoms are dots and the bonds or lines just like you draw on chemistry class on the whiteboard, you can draw that structure. Can you look at it? And can you predict what it's going to smell like? And the answer, except for some really basic smells, the answer is no. The answer is if you move one bond over, one pair of electrons somewhere else, you can go from roses to rotten eggs. And that starkness, that non-linearity in medicinal chemistry is called an activity cliff. And that's what makes the problem hard, is little tiny changes can take you from a molecule that is so popular that it actually forms memories of what cleanliness is.

Like the smell of dryer sheets, that kind of powdery smell, it's the smell of Lily of the Valley, that's in some countries and in some times is one molecule, Lyral. But if you just move one bond over, it's totally odorless. Totally useless, from a commercial-

Danny Crichton:
Wow.

Alex Wiltschko:
... perspective. So very small changes make huge differences to the smell of a molecule. That was the mystery. Now the thing that had come to pass a little bit before, maybe 2, 3, 4 years before I started the group, was machine learning in the field of chemistry started to really work. It started to be something that you could build a research program off of. And that was because of a lot of very hard and very deep thinking by some folks from Harvard, a former mentor of mine, Ryan Adams, Alan Alan Aspuru-Guzik, and then also a lot of Googlers, Steve Kerns, Pat Riley, and then David Duvenaud, who spent some time at Google as well.

And they made machine learning work for chemistry. And that was actually a really big thing. And this is a very strange thing to have to be obsessed about, but machine learning didn't work on molecules, because machine learning really was only working on things shaped like rectangles. So machine learning was really good at things that were shaped like a grid, images, really good at things shaped like beads on a string, which is how we think of language, and really bad or just doesn't work on things that had arbitrary shape and molecules aren't grids.

The general way that we talk about objects that are structured like molecules is graphs. Maybe a couple years before we started working on smell, graph neural networks started to work really well. So that was the breakthrough that we capitalized on, that we built on top of in the group, which is to say, "Let's use graph neural networks, which have already been shown to work in areas of drug discovery and in chemistry. Let's try them for smell." We were able to train neural networks that accurately predict what things smell like, and we could do so at state-of-the-art levels. And actually a paper that is in review right now, it turns out we could do it at superhuman levels.

Danny Crichton:
So covering more smells, more domains than our 350 or so nasal channels.

Alex Wiltschko:
More reliably. So what we did is-

Danny Crichton:
More likely.

Alex Wiltschko:
... we actually predicted the smells of molecules that had never been smelled before, so completely new. Some of them had never been made before. And we kept our predictions secret, the neural network predictions. And then we had a panel of people that we'd trained to rate these odors reliably, smell the molecules and tell us what they thought it smelled like. Now each person, individual on the panel, can be good or bad at that task, but the average, the collaboration of those people is much better.

So the question is our neural network worse than the worst person in the panel? Is it better on average? And it turns out our software is better than the average panelist most of the time., Which means if you're going to add a new person to the panel, you'd actually prefer to add our software program than to train up a new panelist. And we call that superhuman odor prediction performance.

Danny Crichton:
There's been a lot of popularity around GPT technology, specifically ChatGpT and LLMs, which are focused on language, and you're sort of taking input of letters and characters and outputting others letters and characters. But you had a very different problem, which was chemicals. Presumably, you had to translate that somehow into a data format that actually could be processed by a computer, and then you had actually do something to actually create the map. So how did you go and build the model itself?

Alex Wiltschko:
Well, I'll describe how we did it when we first started, and of course, we're constantly upgrading our systems and taking advantage of the newest technology. But the way that we did it is we took molecules and we encoded their graph structure. So we looked at which atoms were present and which atoms were connected to which other atoms. And we fed that as a graph into the algorithm. That's what makes graph neural networks special in this case is they can accept that kind of input. And then the output was a description, an encoded description of what it smelled like.

So we had a 100 or so different odor descriptors, and so we had a 100 dimensional vector and most of the entries in this list were zero. Meaning, zero cinnamon, doesn't smell like cinnamon; zero apple, doesn't smell like apple. And then some were one, one pear, smells like pear; one vanilla, smells like vanilla. And usually there was 2, 3, 4, 5 labels active and the rest were inactive. So we're trying to predict this sparse vector as an output from the graph structure as an input. And with many thousands of examples of graphs as input and this sparse odor label as an output, we can actually learn the patterns that relates the structure of a molecule to its odor, and we can do so successfully.

Danny Crichton:
And one of the things that I'm curious here, when we talk about smells, and you had talked earlier about your first perfume that you had experienced. Obviously, these smells are combinations of smells, they're sub smells that are built-

Alex Wiltschko:
Absolutely.

Danny Crichton:
... into this. You were talking about kind of the different phases. You kind of go through this kind of Shakespearean play that happens as you smell it, different phases, and different parts get recognized. When you think about from the AI technology, how do you sort of break that up? So if I were to literally just spray a perfume and try to capture what's happening there, are you just sort of capturing all the different subcomponents or how does that work?

Alex Wiltschko:
That's a great question. So you never ever, ever, ever smell one molecule at a time. And so this first step that we took scientifically was a pretty gross reduction of the problem because you always smell mixtures. You don't just smell one molecule. Which is, we think of as a graph in our machine learning software, we smell a set or a bag of graphs, a bunch of molecules all mixed together. And they don't all come off your skin or off your hair, if you've washed your hair with soap. They don't all appear in your nose at the same time. There's a time course to them. That's really complicated.

And frankly, it's still a frontier. It's still an area of artificial intelligence that nobody has, in public, demonstrated a real ability to model. And it's something that we think about very hard and I'll have more to say in the future about how we tackle this and what we can do here. But suffice to say that the mixture modeling problem is an open frontier of both neuroscience and also of artificial intelligence. And it's something that we're very keen on.

Danny Crichton:
Curiosity, is that unique to smell itself or are there other applications that could have mixture modeling problems show up?

Alex Wiltschko:
The one that we think of as well is insect repellents. So when an insect wants to come find you and bite you and maybe give you malaria, and there's a million people a year that still die from malaria, mostly children, mosquitoes are sensing mixtures too. And it sounds strange, but we don't really know how DEET works. DEET is our best defense against mosquitoes, but it's getting worse over time, because mosquitoes are becoming resistant to DEET. They're becoming insensitive to it. So we need better insect repellents.

And the mixture modeling problem is important there too. We also design new insect repellents with the same technology that we use to design new scents. And we do that with the Bill & Melinda Gates Foundation, and they've been a wonderful partner. The problem of modeling how that molecule will function on the skin, because your skin is also giving off your own scent, but a part of the problem here is to model how a mosquito will perceive that bouquet, that blend of molecules of the eau de you, and the repellent that is placed on top of your skin that's supposed to send the mosquito away. So that's a very, very important public health problem.

Danny Crichton:
We started the show off talking about California Adventure and this ride at Disneyland, you have this orange smell that comes on. We have a ton of products in our lives, shampoos and perfumes that offer these particular smells. And there's this translation that we talked about, which is you're going from the smell to memory. That you have this kind of direct path from your nose straight into your olfactory bulb, straight into your brain. That makes it very, very powerful.

So I guess the last question here as we sort of finished this episode up around AI is, what is the stage beyond you're able to identify a smell, you're able to reproduce it, you're able to reliably connect it to a human, how do you connect it ultimately to memory? That seems to me like the last step. Is there a way to actually start to predict like, okay, here's a molecule that you've never seen, but given what we know you like, and I'm thinking like vineyards and winos and enologist, if you want a better word than whinoes. Because [inaudible 00:25:24]-

Alex Wiltschko:
Enologist is the fancy whino.

Danny Crichton:
Wino, a fancy whino, a little cookie in between each one. How do you start to connect into that? Because I think that's a huge question around the consumer side of this is connecting into people's memories and experiences around smell.

Alex Wiltschko:
I think we let people do it themselves. In the same way that a 100 years ago, everybody started to pick up a camera. Everybody started to be able to control and to capture, and, frankly, to freeze time, to capture a visual moment, to see a sunset and to hold it and have it forever. That was something that didn't exist 200 years ago, that came into existence, got perfected, it got manufactured, and it got distributed. I think that that's coming for smell. It's not so much can we identify this pattern and tie it directly to your memory, but while you are making the memory, can you capture it? And can you hold it? And can you replay it? I think that's the question.

Danny Crichton:
And that's the question that you are going to try to answer in the years ahead. But Alex of Osmo, thank you so much for joining us.

Alex Wiltschko:
Thank you for having me, Danny.