Generating "hunches" using smart home data 🏠

Smart home data is complicated. There are all kinds of devices, and they are in many different combinations, geographies, configurations, etc. This complicated data situation is further exacerbated during a pandemic when time series data seems to be filled with anomalies. Evan Welbourne joins us to discuss how Amazon is synthesizing this disparate data into functionality for the next generation of smart homes. He discusses the challenges of working with smart home technology, and he describes how they developed their latest feature called “hunches.” Sponsors: O'Reilly Media https://www.oreilly.com/changelog – Learn by doing — Python, data, AI, machine learning, Kubernetes, Docker, and more. Just open your browser and dive in. Learn more and keep your teams’ skills sharp at oreilly.com/changelog https://www.oreilly.com/changelogChangelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Evan Welbourne – LinkedIn https://www.linkedin.com/in/evan-welbourne-51004b31, X https://x.com/evanwelbourneChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Video about hunches https://www.youtube.com/watch?v=whahElqS5eA&feature=youtu.beAmazon’s Alexa can now act on its own hunches to turn off lights and more https://www.theverge.com/2021/1/25/22249044/amazon-alexa-update-proactive-hunches-guard-plus-subscriptionWhat Are Alexa Hunches? https://www.amazon.com/gp/help/customer/display.html?nodeId=G7F5F7K93GKSLC4FUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  When we think of smart home at Amazon and within Alexa, we're kind of thinking if we're really going to add intelligence to the home, what can it really do for customers? And how do we make it really easy? So one part, of course, is like, okay, we're going to get Alexa into the picture. So now we have voice control of the home. It's kind of like the great enabler, the great simplifier. You can just say, Alexa, turn off my light and she'll turn off the lights. Very, very simple type of interaction. But the other sort of piece of that that we think about a lot is what we call kind of an actually smart home, which is it's not just sort of a fancy remote control for your home. It's not just sort of like I push this button and then that automation happens. It's having a home with this intelligent assistant Alexa that can really do things on your behalf, things that are really valuable to you, help you achieve high level goals

• 0:44Speaker 00

  like living more sustainably or just being more comfortable or keeping your family safe.

• 0:52Speaker 01

  Big thanks to our partners, Leno, Fastly and Launch Darkly. We love Leno, they keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com.

• 1:11Speaker 01

  This episode is brought to you by our friends at O'Reilly. Many of you know O'Reilly for their animal textbooks and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos and all their conference talks. Plus, you can learn by doing with live online training courses and virtual conferences, certification practice exams and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics, machine learning, AI, programming languages, DevOps, data science, cloud, containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills,

• 1:55Speaker 01

  then check out O'Reilly's online learning platform. Learn more and keep your team's skills sharp at O'Reilly.com/changelog. Again, O'Reilly.com/changelog.

• 2:12Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:38Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by Chris Benson, who is Principal Emerging Technology Strategist at Lockheed Martin. How are you doing, Chris?

• 2:54Chris Benson?

  I am doing fine. Just been like crazy busy. We have this beautiful spring weather and you know, I don't know if I've mentioned this before, but my wife and I are planning to move. And so we're thinking about new house and we're going to build on a five acre plot of land. And you know how I am with all the animals I love doing wildlife rehab and all.

• 3:14Daniel Whitenack?

  Following the trend of people getting a little bit further out after COVID. I know there's a lot of people sort of moving out of various places like San Francisco and other places after working remote for some time and all that.

• 3:28Chris Benson?

  Yeah, the price of wood and lumber for building is it an all time high. And so we're trying to figure out what our, we could do it soon, but we're trying to figure out timeline. But I'm having a blast trying to think about what I want in my new home. And as I'm trying to think about like-

• 3:43Daniel Whitenack?

  Is it going to be a smart home?

• 3:45Chris Benson?

  You know, I have been bringing that up with my wife quite a bit in terms of like, you know, what could we do? And she, of course, I bring it up and she rolls her eyes at me because it's the kind of thing she expects from me. But I'm, I'm pretty excited about this idea of a smart home. We're building from scratch. We can do all sorts of cool things. And, and I need some new ideas, my friend.

• 4:05Daniel Whitenack?

  Yeah, I mean, we've gone, you know, locks, thermostat, some other things. I know at my wife's business, they have a bunch of, you know, Alexas around the work because they play various music around and, you know, they have them in the offices, actually. So, yeah, I mean, there's a lot of choices. Of course, you could go, I think there's the like meme from Silicon Valley where the, you know, they have the smart fridge and, you know, they all hack into it or something and display all sorts of profane things. But yeah, I think you can, there's so many choices out there now. And I'm excited because I'm definitely interested in that topic. And today, we we've got a chance to to talk a lot more about it with Evan Welbourne,

• 4:48Daniel Whitenack?

  who is, who leads the smart home machine learning team at Amazon. So who better to

• 4:54Chris Benson?

  I'm looking forward to some ideas here.

• 4:56Daniel Whitenack?

  to inform you about your new house, Chris.

• 4:56Chris Benson?

  Yeah, absolutely. Welcome.

• 4:58Speaker 00

  Thank you so much. So glad to be here.

• 5:00Daniel Whitenack?

  Yeah. Well, before we jump into all that good smart home stuff, could you just give us a little bit of information about yourself and your background?

• 5:11Speaker 00

  Yeah, sure. So my background is actually focused primarily on the Internet of Things. And I've worked in that space for, you know, since the early 2000s, when I was a grad student at the University of Washington studying computer science. And at that time, I was working on things like sensor networks, RFID, you know, GPS, some of the really early smartphone stuff. And a lot of that work was trying to infer the context of the customer or the user from like, you know, all that diverse sensor data, and for things like running versus walking, or are they at home or work, are they commuting, and then trying to build and evaluate applications using that type of intelligence. So things like activity trackers, reminders, various types of assisted living, location-based, you know, social networks, things like that. And so machine learning was always a tool.

• 5:57Daniel Whitenack?

  I'm wondering if that, is that sort of activity tracking technology, is that really? Like a product of some of the sort of neural network, deep learning boom of recent times, or have people been trying to do this sort of activity tracking sort of thing for some time with more or less success?

• 6:16Speaker 00

  Oh, yeah, it's been going on for quite a while, in research at least, you know, 20, more than 20 years, people have been just experimenting with simple things like, you know, accelerometers or motion sensors and then can we use, you know, I was using decision trees quite often, you know, just the most basic of models. You can get pretty far with basic models, but of course, when you're really trying to scale something and make it work for everyone, you end up, yeah, you've got to use the more sophisticated approaches.

• 6:42Daniel Whitenack?

  Yeah, makes sense. So how did that activity tracking sort of work lead into later things in your career and where you're at now thinking about smart homes?

• 6:52Speaker 00

  Yeah, so sort of by the end of my PhD, it was clear that machine learning wasn't just a tool, it was kind of like the tool, right? So a lot more focus on machine learning. And then at that time, smartphones, you know, the iPhone had just kind of come out and it was becoming clear like smartphones were probably going to be the technology that would kind of carry us to the next wave of internet of things and start, you know, we'd start realizing some of this vision. And so I went from graduate school, I went to work for Nokia Research, which was still at that time, like the, you know, the big mobile phone company in the world. And I worked as a scientist there for a few years. And then I went to Samsung Research and I led the device intelligence group there. And that was, you know, very much similar kinds of work, you know, a lot of on-device machine learning, like running algorithms on the phone or on a wearable to infer fitness activities or, you know,

• 7:37Speaker 00

  learn preferences of the customer to sort of recommend content, that sort of thing. And then kind of fast forwarding from there, you know, I've more recently come to Amazon, attracted mostly by what felt like the next big wave in consumer IoT, which is smart home. And so as you said, yes, I, at Amazon, I've been leading the, the smart home machine learning team.

• 7:56Chris Benson?

  You know, it just is a start because I suspect most listeners kind of think they know what it is. But I'm curious if you can tell us what how you and Amazon think of a smart home. What is it, you know, just to get us all on the same idea of what that phrase means? Because it's been marketed about over the years. What is a smart home when you work at Amazon?

• 8:14Speaker 00

  That's, that's a great question. I think there's various ways to answer the question. I think to do justice to the area, I have to start with saying that smart home, really, it's been around since at least the 80s.

• 8:24Chris Benson?

  Yeah.

• 8:25Speaker 00

  Smart home technology, there's like X10 networking, there's devices that are connected and you can kind of program your home. That stuff's been around for years and there's been a lot of DIY folks that are just really heavy into that and have they've been developing the technology for years as well. So that's sort of one slice of smart home is that really sort of techie DIY, like wire up your devices to automate your home. A lot of that's about home automation.

• 8:47Chris Benson?

  Yeah.

• 8:47Speaker 00

  And I think when we think of smart home at Amazon and within Alexa, we're kind of thinking, you know, or of course, we're sort of accepting that history and also sort of looking forward about, well, what can a home, you know, if we're really going to add intelligence to the home, what can it really do for customers? And how do we make it really easy? So one part, of course, is like, okay, we're going to get Alexa into the picture. And so now we have voice control of the home. It's kind of like the great enabler, the great simplifier. You can just say, you know, Alexa, turn off my light and she'll turn off the lights. Very, very simple type of interaction. But the other sort of piece of that that we think about a lot is what we call kind of an actually smart home, which is it's, it's not just sort of a fancy remote control for your home. It's not just sort of like I push this button and then that automation happens.

• 9:31Speaker 00

  It's having a home, you know, with this intelligent assistant Alexa that can really do things on your behalf, things that are really valuable to you, help you achieve high level goals like living more sustainably or, you know, just being more comfortable or keeping your family safe.

• 9:45Chris Benson?

  Could you kind of give some examples of how you might implement some of those ideas? And it can be anything you want, but I'm just trying to wrap my head around it. Real or a vision, maybe? Yeah, it doesn't have to be something that you've done yet. I'm just curious kind of what's in your head. Like as we record this, we're still in our houses in the late, hopefully, pandemic period and stuff. What might I be doing going forward with that? Like, what could I do now? And what might be something that you're thinking in the near term? We'll talk about the distant term later on.

• 10:12Speaker 00

  Yeah, yeah. So I think there's kind of three things come to mind. There's sort of three forms of sort of control or interaction with a home. I think it's a useful frame to think about. And one is what we call directed control. And that's where we're explicitly controlling a device with, you know, literally remote control with an app or with a voice command. So that's where I would say, you know, Alexa, turn on the lamp and she turns on the lamp, or you can control groups of devices, all of that. So that's kind of one mode of interaction that can sort of simplify the management of the home and kind of an everyday experience. And then we have, as I was describing, there's kind of this more classic mode of smart home, which we think of as kind of programmed control. And this is where the customers kind of pre-s specifying procedures that they want to happen using

• 10:53Speaker 00

  a program. The customer is thinking about kind of signals, logical conditions, you know, the actions that they want to happen. Maybe like shutting down your house at night. I say, Alexa, good night. And then all my lights shut down, the temperature goes up, like all the things that I want to happen will happen. So we've sort of exposed that type of interface as well with our routines products. A lot of people really are excited about that product as well. And sort of more recently, we're also thinking about, well, okay, so what's sort of the next sort of actually smart experience? And we're thinking of this kind of mode where it's more intelligent control. And this is where we're trying to further simplify the experience for customers by having Alexa more autonomously manage their home. So here Alexa is going to have what we think of as algorithmically derived

• 11:37Speaker 00

  intuitions or hunches. We'll probably talk about that. That's the name of a product, Alexa hunches. And then the customer, they really just have to focus on their own life activities and kind of think about their high level goals, like living more sustainably. So Alexa may have a hunch about, well, you might want to turn down or turn off the light in the basement. It looks like it's on anomalously, or you know, she'll automatically turn down your thermostat to save energy while you leave home. Or you may have the goal to help keep your family Safer, so Alexa might have a hunch that there's a door downstairs that looks like it's usually locked at this time. Maybe you forgot to lock it. Do you want to lock the door? And then she'll lock the door on your behalf. So kind of starting out with simple things, but if you look forward, I mean, there's all kinds of things that we can do on behalf of customers in a smart home.

• 12:19Speaker 00

  If you think of, you know, Alexa is really your personal assistant who has this kind of superhuman power of knowing all about things like energy consumption and how to reduce it. Or, you know, observing at any one time how the status of your locks or your security system and all the other sort of devices and sensors in your home that Alexa can manage.

• 12:37Daniel Whitenack?

  One of the things that I've been thinking about more and more, partly based on some of the conversations we've had on the podcast, but partly based on some of the work we're doing is human perception around AI technologies, especially when they're introduced in a new sphere. So one of the things that I've realized kind of over time is, for example, if you think about the smart home, I think there's a lot of people out there that are used to having a dumb home, right? And so you're introducing like this whole new way of thinking about your home and devices in your home. And essentially trying to introduce a new framework for people in terms of how they interact with things that they aren't used to interacting with in that way.

• 13:20Daniel Whitenack?

  I'm wondering if you have any sort of perspective or learnings on that front in terms of where are we at on that sort of human interaction spectrum with smart home technology in terms of maybe younger generations, older generations, or maybe geographically where people are sort of catching on to this technology more quickly and where the challenges are in terms of helping people adopt this sort of technology in a place where they're not really used to having a smart device.

• 13:51Speaker 00

  That's, it's a great question. There's a lot there. I think there are many challenges, given sort of the diversity of customers that are now interested in smart home technologies, right? It's the long history includes lots of really tech savvy people who are going to make it work for them one way or another.

• 14:05Daniel Whitenack?

  Yeah, yeah, there's kind of a group that's like, I really want a voice assistant and I'm gonna, even if it's hard to use at the beginning, I'm gonna push through and I'm gonna, like, I'm gonna use it, right?

• 14:15Speaker 00

  Exactly, yeah. Those are always, you know, the super valuable early adopters that are kind of pioneering these new technologies. But of course, right, we've got all kinds of other customers that are sort of less savvy. Their interest may not be programming their home and wiring up all these devices. And it may be more about just keeping their family safe. Or like, you know, maybe, you know, I've heard about smart home is kind of interesting. I'm not too confident about setting it up and using it, but it just seems pretty cool. I want to give it a try. And so it's interesting. That's one of our key challenges is trying to help customers through that journey, right? From getting their first light bulb to being the kind of fully enrolled customer that has maybe multiple devices and is taking advantage of some of these kind of higher intelligence features. And so that is one of the things that my team focuses on is kind of facilitating

• 15:00Speaker 00

  that journey with customers, helping them set up devices intelligently and kind of helping them recover from really common kinds of errors that they run into when they're setting up and starting to learn how smart home works.

• 15:22Speaker 02

  Change log plus plus is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and help sustain our production of practical AI into the future. Simply follow the change log plus plus link in your show notes or point your favorite web browser to change log.com slash plus plus. Once again, that's change log.com slash plus plus.

• 16:07Chris Benson?

  So I think that was a great segue there in terms of what you were just describing to talk about. I'd love to talk about how you're really putting the machine learning kind of at the central enabler of being able to make all this happen for people to where they're able to do that. I know you mentioned earlier that there is a service called hunches that you guys are working on, and I would love to understand more about that because I think I've been waiting for something like that for years. I'm selfishly thinking about the new house that we're going to build and how that fits in, so I can't wait to hear this.

• 16:39Speaker 00

  Yeah, so yeah, hunches is a great example of sort of a new. Paradigm within smart home at least. And the idea of a hunch is that it's an algorithmically derived intuition that Alexa has. And so, you know, for those of us who listen to this podcast, you know, we were using a model to make a prediction and there's a confidence score associated with it. And then there's this sort of designed experience around it that kind of supports that type of prediction. It's sort of a prediction with a confidence. We may not be 100% right. We accept that sometimes Alexa is going to be wrong, but it's a hunch, you know, like Alexa has an idea of something that might be helpful to you. And then there's this kind of collaboration with the customer around not just, you know, do we take an action in the case of a certain hunch?

• 17:20Speaker 00

  But how do we learn more about what the customer's intentions are and what their goals are through this sort of series of hunches each time we request feedback from the customer and we learn more? There's probably, there's kind of a few distinguishing characteristics to hunches. One is that, as you can imagine, they're personalized to the customer. You know, they're always about some particular device in their house and maybe corresponding to some behavioral pattern that Alexa has observed. They're also dynamically adaptive to the home and to the customer's current context. So, right in the example of the basement light is still on and it's like nighttime, it looks like a time when you might go to bed, Alexa may sort of reach out and let you know that your light is on and ask if you wanted to turn it off. Or in some cases, you know, we can talk later, we also have automatic actions for hunches where Alexa would just automatically turn it off.

• 18:04Speaker 00

  But that's after we've sort of built a little more confidence with the customer. They're also, as noted, they're non-deterministic. So, you know, we're not going to deliver a hunch, we're not going to take an action unless Alexa has high confidence. So that model has to be pretty confident across all of the signals it observes that this is something the customer would actually value before we actually surface it. And then finally, as described, it's refined in the loop with feedback from the customer. So every time we're delivering a hunch, we're inviting the customer to give us feedback, often explicitly, you know, Alexa may ask, did you want me to turn off that light? And they can say yes or no, or, you know, they can give feedback through the app. But sometimes there's implicit feedback. So if we go and we lower the thermostat, we'll kind of watch to see if they turn it back up later. And that's another kind of feedback that we can constantly learn from.

• 18:49Chris Benson?

  It sounds like that feedback, is it the central mechanism for establishing trust with a customer for the new service? Because we have so many conversations with people who are doing these amazing things with machine learning and so much of it now is advanced, is requiring people to make that mental shift. And we've talked a little bit about that already. But is that establishing trust so that you feel you can incorporate hunches into your life? Like, what else do you guys think about in terms of how to get there and what the next steps are? Because obviously, the trust has to be a huge part of the strategy on moving this all forward.

• 19:25Speaker 00

  Absolutely. I mean, feedback is absolutely its core to this idea of hunches. We think of it as a collaboration with the customer. We're earning trust continuously as we sort of get feedback from the customer, learn, personalize, adapt to their patterns. I think a few other things just about the UX or the CX, as we say at Amazon for hunches is the kinds of hunches we started with a couple of years ago were really sort of these extemporaneous delighters, like the case of like reminding you to turn off your light or lock your door. They don't happen that often. They're sort of targeting anomalies, but they're sort of, it's kind of a delightful experience when they happen. Fortunately, we've been able to kind of tune the models so that we're usually right about those anomalies. The customer does want to turn off the light or lock the door. And so like that sort of spark, even though it's like kind of a simple thing, that spark, it really earns a lot of trust.

• 20:10Speaker 00

  And then the customer also feels in control because they have that feedback and they can say no if it was not the right thing and we're not going to ask them again in that scenario. That definitely kind of helps us build that trust as we go forward. And then absolutely, like, I mean, the way we've sort of continued to work on the hunches product is to incorporate gradually more and more use cases. And then most recently, just this last year, we launched the hunches automatic actions where we're working with the customer in advance to sort of help them understand that Alexa's now going to be able to take action and adjust your thermostat or turn off your lights based on her inferences about what's going on in the home. And you can sort of consent to that in advance and then Alexa will turn off your lights if she thinks you're asleep.

• 20:53Speaker 00

  So that kind of gets that sort of other element of that earning trust, which is explanations for the inferences. So if we have a hunch- That we should turn off that light. That'll show up in what we call the hunches dashboard within the app. That shows these are the actions that Alexa took last night or the last 30 days. And you can see, you know, a very simple explanation for each one. Alexa turned off your light because she thought you were asleep, or she turned on the thermostat because she thought everyone in your house was away. And then you can give feedback right in there, but you're understanding more about how Alexa's working.

• 21:22Daniel Whitenack?

  Yeah, I'm curious, just on the practical side of things. I'm assuming as more and more smart devices are integrated in people's homes, I'm thinking about the data side of developing something like hunches. It seems to me like there's all these different customers that could have all sorts of different, unique combinations of devices in their home. And so like the data is not the same for customer A versus customer B. And maybe the history of that data is not the same. And there's also like geographic factors or lifestyle factors. So how do you even, from your team's perspective, do you have any good, maybe workflow hints or tips for people that are dealing with this sort of complex data situation?

• 22:05Daniel Whitenack?

  And really exploring that data and getting down to the, because you have to start somewhere, like you were saying, you're building in incrementally more and more of these. Any tips for people out there that are maybe dealing with this sort of complicated data situation? And trying to get down to where should I start in terms of creating value? Because there's so many different varied ways that I could go about this.

• 22:29Speaker 00

  Absolutely. Well, it's very, very insightful. I mean, that touches on kind of a really key challenge for smart homes. Just one part of that is the challenge of providing that sort of consistently high quality inference across customers. They have so many different types of homes, so many different types of devices. They use them differently. They live in different places. I mean, it's a super hard problem. And there's a few, sort of, maybe I think, sort of, three things. I mean, one, personalization is always useful in the early phase. We may not know instantly about your house, or we may not know about everyone's house, but we can learn how you use your bedroom reading lamp. That's one thing where we can sort of learn over time, some basic things about that, just on your data alone.

• 23:09Speaker 00

  And then, yeah, secondly is if you are able to, if you're empowered to kind of design a feature, including the kind of user-facing interaction, then it really helps to build that feedback loop right into it, right? Just like hunches, we propose a hunch and then we get feedback on it. And that's great for gathering training data from the only people who can really label it about the context of their home. The end customer is really the main person who can provide you that most accurate label about what they want to do. But the, sort of, the third area of work, I think this is really where I see kind of the, one of the key scientific challenges for smart home, is trying to infer information about the home, like trying to infer activities in the home across such sort of varying data sets from individual customers. And it's actually pretty sparse data from an individual home,

• 23:54Speaker 00

  you know, someone, we're just looking at the lighting data. People just turn on and off lights four times a day, five times a day. You know, you gotta, you gotta wait a while to get data on any given house. So you've got to find ways to sort of learn across customers, not just, you know, you probably start with personalized models, but there's a lot of investment we've made in sort of deep models that are trained across millions of different customers. And there's even, you know, I could talk more about it, but there's even some, I think, fundamentally new scientific discoveries about what's possible there, like how much similarity there is between customer behavior in one home and a hundred or a thousand other customers.

• 24:32Daniel Whitenack?

  Yeah, and is that like, because I know a lot of the devices these days, like, people might have a mix of smart home devices in their home that are from different brands even, and report sort of different sets of data. Is that correct?

• 24:47Speaker 00

  That's right.

• 24:48Daniel Whitenack?

  Yeah, so like, things are coming in with different... I guess different feature sets, different things that are represented by different brands and maybe even different formats. And maybe there's some standards around that now. Is some of that even just like synthesizing some of that data together and figuring out how data from different brands of different devices, reporting different data is sort of matching up to, hey, this is lighting information over here and this is lighting information over here and it's reported slightly differently, but we can, you know, how much work is there in sort of synthesizing that across all the very devices these days? Or is it more standardized than I think?

• 25:28Speaker 00

  Yeah, that's a good question. So it is a little bit standardized. You know, we have an API that partners use to report any kind of data like that. That's certainly helpful. Of course, the reality is like, you know, across many different kinds of companies, partners, devices, the quality of the data varies and certainly the content of the data varies. There's more than, I think, there's more than 140,000 different types of devices that connect with Alexa today. And so that's a lot to keep track of.

• 25:54Daniel Whitenack?

  No big problem.

• 25:55Speaker 00

  No big problem, yeah. There's a lot of different data there. And so it is true, there's a lot of variation across partners. There's a lot of variation across device types. And there's not really any one universal solution to just immediately cleaning up all data of every type, unifying it all into one big model, but there's sort of categories of data. A lot of it's time series data, of course. And so you see these kind of point events of like a light turned on or change brightness. That's the same for all different kinds of lighting. So you can kind of develop sort of a model or understanding of lighting. Thermostats are a different picture of security systems are different. Smart plugs, they're kind of similar to lighting, but that, you know, they also have different types of usage patterns. You might want to model them a little bit differently. So you tend to start

• 26:40Speaker 00

  looking at devices and categories and standard practice, you might want to build a layer kind of above that raw data to just kind of, if you have lighting devices, for example, you may want to build a layer that tries to sort of smooth out any kind of noise in the data and offer you, for example, like the state of the device with a confidence score rather than just the raw information.

• 27:01Daniel Whitenack?

  Okay, yeah, so you're kind of, you're building that sort of middleware layer that does some sort of synthesis or correlation of that data together, smoothing and that sort of thing with confidence scores. And then I would guess that like something like hunches would then rely on the fact that, hey, I know about lighting data and the trends of lighting data generally. So I'm guessing that that would help with something like that. Is that right?

• 27:28Speaker 00

  Yeah, definitely. And you know, we do, for the kinds of models we use in hunches and most of our products, we do try to incorporate as much information as we can, not just from this sort of time series type of data, but even just kind of the metadata about devices. So if we're, again, if we're talking about lighting, customers are able to name their lights. Sometimes they just leave it with a default name like first light, second light. But often they name them, you know, living room lamp or reading lamp or basement lights, you know, what have you. And even just in the name of the device, there's a lot of information. We know that something is a bedroom lamp. It's probably going to be on in the evening for an hour or two and then it's going to be off overnight versus a front porch light. Often they're on all the way overnight and they're off during the day. There's a lot that we can kind of pull in to kind of add to the model.

• 28:13Speaker 00

  And then by extension, of course, if you think about sort of training models across millions of customers and millions of different devices, you can start building kind of like device embeddings that kind of distill all of that information about not just, you know, is it a bedroom lamp, but sort of like what kind of bedroom lamp is it, you know? There's a lot that you can kind of pull together between the behavioral patterns and then the names, the devices, the way customers interact with them.

• 28:36Chris Benson?

  So you were saying something a few minutes ago that my brain's been spinning on a little bit. And I wanted, I want to go back and do a kind of a late follow up on it. When we were talking about, you know, models that could generalize across thousands and millions of people in terms of these activities, you know, the turning the lamp on and stuff like that. As we go forward in time, And you're moving ever more presumably into kind of personalization and really not just supporting all of our lives, our lifestyles, maybe the right word for it, but starting to hone in on, you know, what does Chris need, you know, what, what does Daniel need and our own differences and stuff. How do you approach that from the sense of you have these tools that you've built that can

• 29:17Chris Benson?

  handle these activities or tasks in a large sense, but over time, I might be going through the house and turning on lights in a different way Daniel does because of some quirk of my own personality and, and you're having to tie my activities into those otherwise kind of mainstay thing, you know, light on, light off, you're recording that, you know, that that's happening. How do you think about personalization as you're moving into that world where it's not just about turning the light on, it's about why Chris would do it versus when Daniel does it versus other people. How do you think about a future where that's going and approach it?

• 29:50Speaker 00

  Right. Yeah, I think that's really interesting. I think part of that, if I'm understanding you correctly, part of that is kind of understanding and perhaps modeling the customer's intention.

• 29:59Chris Benson?

  Yes.

• 29:59Speaker 00

  What's, what's actually going on here? Not just, you know, I think that specific light is gonna turn on.

• 30:04Chris Benson?

  You said, you said it much better than I did, actually.

• 30:07Daniel Whitenack?

  Yeah, I know in like in a chat dialogue sense, there's like the idea of user intense. I don't know if there's the idea of like user intense when you're looking at smart home data, like, hey, they're doing this on, you know, they're doing this on a Saturday versus they're in there doing this on a workday or something.

• 30:27Speaker 00

  Yes. Yeah. So the, the short answer is yes, at many levels, we have that concept and then we approach modeling of customer intents in different ways. So one, one thing to note, of course, even just in the most simple kind of directed control scenario, we have a concept of what was the customer actually trying to do? What's their intention? Did, you know, we can sometimes see that they're trying to turn on a light and perhaps they didn't get the name of the right of the light correct and it failed, but we can still tell that their intent is to turn on that particular light, right? So there's, there's just very simple kind of intentions, but to get into kind of like activities on a, on a weekday or Saturday and, you know, what would they want to happen at this particular time? That's where it starts getting really interesting. I feel like.

• 31:07Speaker 00

  That problem, I think of it as modeling the context of the home and the customer. That's where the sort of ideas like the activity of the home, the activities that the customer is engaged in. Those are sort of a key part of that picture, like understanding that they're having dinner now or that they're all asleep or that they're away from home. Whatever the relevant activities are is a big piece of that. The other way I think of that, and the other thing that's important and we're just sort of getting to in the scientific sort of agenda for our team is really understanding how to help customers achieve their long-term goals. So not just, you know, turning on a light or not even just controlling their thermostat in a way over the day that makes them comfortable, but helping them actually save money on their energy bill

• 31:50Speaker 00

  next month or, you know, helping them stay safe throughout the year with their, you know, various security devices and security system. And kind of balancing those goals with each other, right? Like if you want to save money, you're going to be biased towards turning lights off quite often, turn the temperature down. But if you want someone to be really comfortable, you might bias towards having the temperature up a little more often, or if you want them to be really safe at home, you also, in some cases, you might bias towards having lights on just in case someone's there. You want them to be able to see where they're going. So it's really interesting problem of balancing these kind of long-term driving goals with sort of the short-term actions.

• 32:28Chris Benson?

  So I know, and as we're talking about this, I know there's going to be at least a few folks out there thinking about what about the security as you're doing this? Because I'm all excited about getting my home, you know, able to basically predict me ahead of time and do that. That's very exciting. There's going to be someone out there worrying about what happens when everything is voice-controlled and you have someone who shouldn't be there and do that. How, how do you think about it? And I really mean security, not in the, in just basic security, but like my voice versus a stranger's voice and, you know, whether there is any kind of recognition built into services or will be in the future as that becomes more of a real life kind of consideration. How are you thinking about that level of personalization going forward where some people should and some people shouldn't?

• 33:13Chris Benson?

  And what's kind of the thinking around that at this point?

• 33:16Speaker 00

  Great, great question. I mean, it's always a lively area of work, the kind of the security and authentication question, hard problems there. The one caveat is that my team doesn't own that space within Alexa, but I can at least comment that, so regarding these sort of security use cases, right, there's, as you may imagine, there's certain use cases we just don't support. Like, you wouldn't want to allow anyone to yell from outside, you know, Alexa, unlock the door. And then we unlock the door. There's sort of layers of security, you know, whether it's sort of voice codes, or there's speaker recognition, right? That's another feature that folks have implemented for, you know, use in all kinds of cases within Alexa. And then another aspect of this really interesting

• 33:58Speaker 00

  is just sort of personalization in general, and how do we accommodate or deny people who are kind of not part of my smart home? You know, like, if someone is a guest and comes to my home, are we going to allow them to control the lights and the music and all of this? For the most part, I'd say that's a very interesting problem. And there's a lot of, you know, it's really a lot of product thinking in addition to thinking about, you know, authentication and privacy, like, what kind of experience do we want to create?

• 34:25Chris Benson?

  You just threw one out that I had not considered at all, and that is, and that would be very different with different customers of yours, I would imagine, is, you know, you have guests over and you said, control the lights, control the music. There would be some people, you know, it's a party, yes, and other people, that's, it's an interesting problem, because there's a lot of nuance there, I think, to tackle, so.

• 34:46Daniel Whitenack?

  Also, no one wants me choosing the music at their party.

• 34:53Chris Benson?

  Note to self, okay.

• 34:54Daniel Whitenack?

  Note to self. So, Chris, you remember, we had the guest on, Nung Ho, from Intuit, who is Director of Data Science at Intuit, and she was talking about, like, pandemic times and, like, time series modeling. And of course, like, time series modeling depends on... You're, you know, how good is the history of your data? And, you know, this last year has just totally blown all that apart for many people. So Evan, I'm kind of curious for the smart home team, machine learning team at, at Amazon. What has that sort of like disruption in the history of data? Meant for your team in terms of maybe new opportunities?

• 35:39Daniel Whitenack?

  There are those things you didn't realize before, but also in terms of like thinking about, hey, maybe we need to do things slightly different to sort of pandemic proof some of our processes. Curious about that, that aspect of your team's conversations over the past year.

• 35:55Speaker 00

  Yeah, it had a huge impact. It's kind of changed everything in the data that we see. Because, you know, all of our data is from the home. And one of the implications of a global pandemic is that a lot of people are staying home a lot more often than they were. Especially, we've got at least this big chunk of the, the early adopter folks who are often tech workers who are probably going to work remote anyway. So one of the things we've seen, for example, as of maybe March or April last year, is that suddenly weekdays started to look a lot more like weekends. You know, people are waking up a little bit later. There's more activity in the home throughout the day. People stay up a little bit later now on, on weekdays as well.

• 36:38Speaker 00

  And to that end, we've had all these models trained on customer behavior patterns across, you know, days and weeks and so on. And we had to sort of switch all those up because what looked typical back in February 2020 is very different from what looks typical in, you know, February 2021. I guess a couple of other notes. One of the things that we've had to do is rely a little bit more on, I guess, again, personalization. We've got to lean into like, what do we think this particular customer is going to do rather than like any customer? Because people, you know, at least within the US states are like opening up and locking down at different times and.

• 37:15Daniel Whitenack?

  True.

• 37:16Speaker 00

  You know, we can't rely on people in Texas, you know, kind of predict what's going to happen in, you know, Montana, you know, it's just different, different places.

• 37:24Chris Benson?

  Yeah.

• 37:25Speaker 00

  So we've kind of leaned in a lot more on personalization to an individual customer. And then we've just sort of reset some of the assumptions the models made about the kind of like, things like trips away from home, for example, you know, there used to be, of course, this very established pattern of sort of nine to five work.

• 37:41Chris Benson?

  Yeah, commuting and things.

• 37:42Speaker 00

  Away from home and come back. And that doesn't really exist anymore. And a lot, a lot of the time now, these sort of trips away from home are kind of short, short trips, like running out to get groceries and coming back, something like that.

• 37:53Chris Benson?

  Yeah, it's, it's true. It's, um, it's kind of funny, just to your point right there. And Daniel knows this, and you may have heard this on a previous thing, I'm taking flying lessons. And I told my wife the other day, you know, because I used, I was commuting, I was going off on business trips and I was often away from home through the whole day and maybe for multiple days. And my life has flipped so much that with my flying lessons and these short trips out, I realized that I'm flying as an amateur, you know, private pilot student more than I'm driving my car at this point, which was a kind of a bizarre realization to do that. But that, that raises another point that I wanted to ask about, and that is, as you're addressing smart home and you're pulling the data from the home, we're also seeing smart technology being implemented out of the home.

• 38:36Chris Benson?

  And, you know, we're, we're getting, you know, automotive, smart capabilities and various other things in our lives that are outside our houses. Any thought into how those integrate over time? And I realize that we're still, you know, you're, you're working your way there and there's, there's a lot of stuff that has to go forward, but at some point, we're going to be moving around in smart vehicles, maybe fully autonomous vehicles, you know, not terribly far into the future at this point, we already have some out there. We're coming home to our smart homes and so much of our lives are being automated and yet they're all somewhat disparate right now. How do you envision that coming together for more of an integrated feel, recognizing that

• 39:20Chris Benson?

  it may not always be just Amazon. Amazon does a lot and you guys may be in a lot of those fields, but if you're looking even beyond that, you have a lot of different players doing different parts of life, if you will. And how does the world come together in an integrated experience that is what that consumer wants it to be?

• 39:37Speaker 00

  Yeah, that's a great, great question, a great observation as well. I mean, it's, you already see some of that. You know, there's Alexa for auto.

• 39:45Daniel Whitenack?

  Yeah, of course.

• 39:47Speaker 00

  And I'll tell you, the way I see it, and also that I think aligned with Amazon's vision is that it's all kind of part of the same problem if you're focused on the customer. Really, we're just trying to help the customer live more simply, achieve their goals. And you know, a lot of that we can do with just the smart home data. But if you pull in data and sort of inferences we can make about what they do in the car, what they do out in the world, you know, maybe with using their smartphone, there's just a lot more you can understand about their intentions and what they want to happen. It's sort of obvious for us, you know, Alexa is kind of the unifying element here. We have this ambient assistant who is able to sort of stay with you, whether you're at home, whether

• 40:28Speaker 00

  you're on the go, you know, with your smartphone. She's kind of present there across these elements of your life and can kind of help tie things together. I think that's sort of the metaphor we're using to sort of unify the data as well as the experience for the customer. I think the opportunities are huge. I mean, as I noted, I spent years working on smartphones and mobile inference. I know there's a lot there to add to the big picture.

• 40:53Daniel Whitenack?

  That's super exciting. I think that's, that's a really good way to sort of tie things up here at the end is thinking about that ambient smart technology sort of permeating your life. I know it's, it's really exciting. There's a lot of challenges. There's a lot of questions ahead of that, but I'm really excited about it. I'm excited to see what Chris puts in his home and we can, we can hear about that.

• 41:14Speaker 00

  I am too.

• 41:15Daniel Whitenack?

  But yeah, appreciate you joining us so much, Evan. This is a really great conversation. And I'm excited to try out some of, some of these hunches and other things and see how they develop over time. Really appreciate the work that your team is doing. So, thank you so much for taking time to join us. Really appreciate it.

• 41:34Speaker 00

  Excellent. Yeah, thank you so much. I'll look forward to listening to the podcast.

• 41:42Speaker 02

  Thank you for listening to Practical AI. We appreciate your time and your attention. If you enjoyed this episode, help us out by spreading the word. Think of a friend, think of a colleague, somebody who would benefit from listening to it, and send them a link. We'd really appreciate it. Practical AI is hosted by Chris Benson and Daniel Whiteknack. It's produced by Jared Santo with music by Brake Master Cylinder. Thanks again to our sponsors, Fastly, Linode, and Launch Darkly. That's our show. We hope you enjoyed it, and we'll talk to you again next week.