How Google is trying to make ambient computing work

Google has big visions for the future of computing. It’s working on building what it calls an ambient computer: a virtual helper that can accomplish anything, anywhere, any way you want. But that vision won’t come easy. Ahead of Google’s I/O developer conference, The Verge sat down with some of Google’s most important executives to talk about the company’s vision, its new hardware and software, and how the company is changing to build the future it imagines. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This episode is brought to you by On Investing, an original podcast from Charles Schwab. Each week, hosts Lizanne Saunders, Schwab's Chief Investment Strategist, and Colin Martin, Head of Fixed Income Research and Strategy for the Schwab Center for Financial Research, bring you fresh insights on what's happening in the markets and why, and what the implications might be for your portfolio. Join Colin and Lizanne as they explore questions like, how do you evaluate corporate bonds that look interesting? And what sectors are on the move right now? Download the latest episode and subscribe at schwab.com/oninvesting or wherever you get your podcasts.

• 0:45Speaker 00

  This episode is brought to you by On Investing, an original podcast from Charles Schwab. Each week, hosts Lizanne Saunders, Schwab's Chief Investment Strategist, and Colin Martin, Head of Fixed Income Research and Strategy for the Schwab Center for Financial Research, bring you fresh insights on what's happening in the markets and why, and what the implications might be for your portfolio. Join Colin and Lizanne as they explore questions like, how do you evaluate corporate bonds that look interesting? And what sectors are on the move right now? Download the latest episode and subscribe at schwab.com/oninvesting or wherever you get your podcasts.

• 1:31David Pierce?

  Welcome to The Vergecast, the flagship podcast of tablet apps that are really just blown up smartphone apps. I'm David Pierce.

• 1:40David Pierce?

  So it's Google I/O this week, which is Google's developer conference and also the company's biggest launch event of the year. There's a ton of product news coming out of I/O this week, and we're going to get into a lot more of that, from the software, to the hardware, to the services, to all the weird stuff Google is working on on Friday's show. But today, I want to look at the bigger picture. Google likes to call I/O its State of the Union, which is actually a pretty good way of thinking about it. Google is this huge, sprawling, decentralized company where teams build lots of things. Often, it seems like nobody seems to know what anybody else is working on. And then, once a year, the company tries to pull everything together and figure out what's Google, the whole thing, the entire company, all these hundreds of thousands of people, actually up to?

• 2:25David Pierce?

  I spent a good chunk of the last week talking to Google executives about that exact question. And it's funny to do that, honestly, because you start to hear the same things over and over and over again. For starters, everybody really likes to talk about Google's mission, which is to organize the world's information and make it accessible and useful. I know this by heart now, because Google people won't stop talking about it.

• 2:46Speaker 02

  You know, our mission in Android is to bring computing to everyone in the world.

• 2:50Speaker 03

  I give Larry and Sergey a lot of credit for coming up with this mission all those years ago.

• 2:54Speaker 04

  We've extended it recently to basically being about being helpful for users.

• 2:58David Pierce?

  But every year at Google I/O, there's the story Google is trying to tell about itself, and then there's the story it's really telling. And this year, the story it's trying to tell is about how powerful its computing tech is. Google talks a lot about the tensor chip in the Pixel phones and how it lets Google do all kinds of powerful AI stuff. It also talks about large language models, which are AI systems that can develop a radically better understanding of text and images and even videos, and how all of that helps Google organize information. Computing and AI, over and over. Computing and AI, computing and AI. And that's all true. But underneath all that, as I talked to folks at Google, I heard a different story. What's going on with Google right now is that Google has become this huge sprawling

• 3:42David Pierce?

  decentralized company where teams build lots of things and nobody seems to know what anybody's working on. And as a result, Google feels kind of incoherent. For so long, you could think of Google as a search company and mostly be right. Now it's, I don't even know how to describe it. It does all the things. And honestly, even in search, it's confusing. There's Google Search, but there's also Google Lens, which is kind of like Google Search, but in a different way. And there's Google Assistant, which you can use for search, but is really a whole different thing. Google is still more or less on the organize the world's information and make it accessible and useful grind. But as the company has gotten bigger, its pieces have stopped sort of making sense together. It makes phones and computers that can barely talk to each other. I mean, dear God, have you ever tried

• 4:27David Pierce?

  to just send a file from your Android phone to your Chromebook? The whole of Google sometimes feels like less than the sum of its parts. That is what Google is working on now. It's trying to take the many parts of Google, the countless features and products and initiatives and experiments and prototypes, and make them make sense together. I think Liz Reid, a VP of Engineering at Google who works on search, actually explained this better than anyone else I talked to.

• 4:52Speaker 03

  A lot of these newer use cases, there's a style or a taste component. There's a way to process information. Some people really find it easy to process video and other people will find it distracting. On the other hand, some people's literacy is not as good. And so a long webpage that you have to read through not only takes time, but they're going to get lost. And a video that's spoken in their language is really intuitive. I think we need to assume that there is no best answer for many of these things. I mean, if there's a height of the Eiffel Tower, okay, there's an answer to that. But if it's like, I'm finding a wedding dress, there is no right answer. People have very different styles. They have different price capabilities as well. So how do we really let you find the dress you want versus your next door neighbor find?

• 5:35David Pierce?

  MultiSearch, which Liz and her team have been working on, is one of the things Google is talking a lot about at I O. And it's a pretty good microcosm of what it looks like for Google to start to put the pieces together. The idea behind MultiSearch is that you can't always express your entire question in one way, right? Google likes to give the example of a dress you like. You see the dress, you want that dress, but you want it in green. So far, there is literally no way for you to ask Google to find that for you. But with MultiSearch, you can take a picture of the dress and then type "in green" and Google understands and processes the search that way.

• 6:10David Pierce?

  One way Google is making all this stuff work is by making it conversational. I heard this also over and over from Google execs over the last week. They're absolutely convinced that rather than you just barking an order at your phone or speaker, that you should have something that more closely approximates an actual back and forth conversation. And what that means is that in the process, Google can ask you questions. It can refine your search results. It can figure things out better than just doing it one exchange at a time. I am skeptical of this. First of all, have you tried having a conversation with a voice assistant? It's either infuriating, or it breaks, or actually, no, those are the only two options. But Google thinks it can work and feel natural and not be the worst.

• 6:52David Pierce?

  And the reason it does is because of those large language models I was talking about. Rather than me try to explain, I'm gonna let Zubin Gavramani, a distinguished researcher at Google, do it. This is kind of a long one, but it's worth it, I promise.

• 7:04Speaker 05

  So about 10 or 15 years ago, the field of language technologies really was revolutionized by deep learning. And of course, at Google, we've been using deep learning very extensively from at least 10 years ago in language and other domains. And then in the last five years, so back in 2017, we actually developed a form of language model called the transformer. And the transformer is in some ways kind of like the grandmother of a lot of the modern language models. So the T in BERT, in T5, in OpenAI's GPT-3, in Meta's OPT, it all stands for transformer. So this has been an incredible breakthrough

• 7:45Speaker 05

  that has enabled a lot of these other interesting language models to flourish over the last five years.

• 7:51David Pierce?

  Okay, let's skip ahead a little. Here's a little bit on how transformers and new language models have changed the way translation works, which is both a big deal and a pretty good explanation of what it means for the whole system.

• 8:02Speaker 05

  You know, the traditional way you think about doing translation is you need a line text, like text in French and English, let's say, to be able to translate from French to English and vice versa, right? That's the way translation traditionally was thought to require. But actually, it turns out with some of these large models, you can just learn to translate in some of these languages using only monolingual text. So the model is exposed to many different languages, but it doesn't need that aligned translation. It just needs some monolingual text without any translation pairs of sentences. And that's the kind of breakthrough that, again, is surprising to me as a researcher who's been in the field for 30 years. Of course, you know, when you dig into it, it makes sense

• 8:47Speaker 05

  because there are patterns in the data that help it extract the common representations between different languages that are needed to do that translation. But it's still a breakthrough that really helps us, for example, deploy, translate across many more languages that are useful to lots of other people.

• 9:05David Pierce?

  Basically, Google is turning practically everything, from text to audio to video to images, into language, and then using these language models to help understand it all. So you can change the way Google understands an image by adding a bit of text, or ask it a paragraph-long question and have it understand you. Here's Nino Taska, who's a Director of Product Management on Google's Speech Team, who had a pretty good idea about where this all goes.

• 9:28Speaker 06

  You can imagine like a 45-minute YouTube video of "do it yourself," steps 1 through 9, and a large language model can actually understand that video. And so now I can ask it to play and be like, "Oh, no, no. Skip to step 4, because I already know 1 through 3." And the model understands that video. And if you go back, that's like how I would talk to you is how I want to talk to my assistant. So it comes back to the beginning of, "Okay, well, what's a natural conversation look like when it's this back-and-forth medium? Not just play a video, and I'm going to sit and be silent while the video plays. I might play the video, and then I might want to interact and talk to my assistant. I might want to jump to a command. I might want to pause, go to the next command." But having this full understanding of the video, or, you know, Lambda can theoretically understand the whole internet, right?

• 10:13Speaker 06

  And so you can ask questions, cross-referencing documents that you just can't today. And when you think about that kind of knowledge that can come into play, this need to have more of a conversational interface just becomes more apparent.

• 10:24David Pierce?

  Like Nino said, most of this isn't possible yet, but this is very much where Google is headed. Step 1 is making Google easier to talk with, which is starting to show up in little ways. The big Google Assistant news this week is that you can talk to the Google Nest Hub Max just by looking at it and then speaking to it. No hot word necessary, because it's using on-device gaze recognition to notice you looking. On the same device, you can also just say, "Set a timer for 10 minutes," or "Turn off the kitchen lights." And Google understands the commands without needing an "okay, Google" ahead of time. And in general, since so much of this stuff is happening on-device, it's both more reliable and lower latency, and generally just much more usable.

• 11:04David Pierce?

  So that's the big-picture stuff. We're going to take a quick break and come back with more on the future of Google.

• 11:14Speaker 07

  You're busy. We know that, so we'll keep this quick. Introducing AMX Corporate, now featuring the Corporate Cash Back Card. For one annual flat fee, get 1.5% cash back on eligible business purchases, plus intelligent expense management with the unmatched expertise and backing of American Express. Built to simplify, designed to scale. Get started at go.amex/ammexcorporate. Terms and cap apply.

• 11:45Speaker 07

  If you're looking to hire, you need Indeed. With Indeed Sponsored Jobs, you can spend less time searching and more time actually interviewing candidates who check all your boxes. Less stress, less time, more results. And listeners of this show will get a $75 sponsored job credit to help your job get the premium status it deserves at indeed.com/podcast. Just go to indeed.com/podcast right now and support our show by saying you heard about Indeed on this podcast. Indeed.com/podcast. Terms and conditions apply. Need to hire? This is a job for Indeed Sponsored Jobs.

• 12:31David Pierce?

  Welcome back. So, the reason I wanted to start by talking about search is that, well, Google's a search company. But one thing that really stuck out to me over the last week is that Google is also increasingly a hardware company. It built the tensor chip to be able to do more of its AI work on devices and has clearly begun to bet big on the pixel lineup in general. And for now, Google execs all over the company acknowledged that practically everything Google wants to do has to start with your smartphone.

• 13:00David Pierce?

  But when Google talks about the smartphone, it actually has a vision that's much bigger than just the smartphone. Here's Rick Osterloh, Google's SVP of Devices and Services.

• 13:08Speaker 04

  Our vision of where the future is is that it's totally different. That, you know, people will interact with devices in a way that's almost like it's one computer. You know, they interact seamlessly across different modalities, different form factors, different use case scenarios, contexts. They'll be able to interact with computers in various different ways. Voice, tapping, and in the future, a lot of things will be done proactively for you. So you don't even really, it's like an agent is doing things for you. And we call this vision ambient computing. We want to bring all of Google's latest innovations in hardware, software, and AI, put it together for a user to offer the maximum possible help we can, be as helpful as we can for users. That vision is one that's very hard to build. Like, it means you have to have expertise across a lot of different arenas,

• 13:53Speaker 04

  both in devices, but also in a number of different technologies in AI and software, and now silicon. Certainly for the foreseeable future, we feel like the most crucial part of that is the pocketable computer, the mobile phone, that continues to be the center of most people's computing focus. And certainly today, the most important element of a computing future.

• 14:15David Pierce?

  Google in general is making this big push away from being a search box you type into and more of an always-on experience, a thing helping you and working with you and sometimes literally talking with you, like the movie Her, all day. And that means Google better make damn sure it's powering the phone in your pocket. The only way this works is if Google is everything and everywhere. Rick showed me the new Pixel 7 and Pixel 7 Pro, the new flagship phones the company is teasing this week but fully launching in the fall. And also the new Pixel 6A, the $449 phone launching now. And Rick said that the key to the Pixel 6A is bringing all that AI, all of that Googliness, to as many people as possible without changing the experience.

• 14:57Speaker 04

  Why I think this is really important is that it takes a lot of the capabilities we've offered for the first time with Tensor on Pixel 6 and 6 Pro, and brings it to a different price point, brings it to a lower price point. $449, beautiful device, you know, it has Google Tensor. There's goods back on the rest of the device. I think people are going to see it for great value for money. But the big punchline is that you get all of the AI power that comes with Tensor on a much more accessible price point.

• 15:24David Pierce?

  Google is also finally launching its own smartwatch. It's called the Pixel Watch and it's a nice looking, round smartwatch running Google's Wear OS. It'll do normal smartwatch things like tell the time and get notifications and work with Google Assistant, but it's primarily being sold as a fitness device with Fitbit built in.

• 15:41Speaker 04

  You'll be able to sync with a Fitbit account. You'll be able to get all of like Fitbit's latest algorithms for health and fitness, be able to track against your personal goals. Everything will sync back to the Fitbit service so you can see it on the web, on your app, and it's deeply integrated throughout the watch face and throughout the device.

• 16:00David Pierce?

  In addition to all that, Google's also making a pair of high-end earbuds called the PixelBuds Pro, which are basically, as far as I can tell, exactly the same as the AirPods Pro, but for Android devices. And Google is even going to make an Android tablet after it spent so many years basically not giving a crap about Android tablets. So what's going on here? Why is Google suddenly going big and wide on all these devices? It had a brief run with smartwatches in the middle of last decade before essentially appearing to give up on the idea. Now it's saying, no, just kidding, we're back, we love smartwatches, Wear OS forever. And same with tablets. Every Android fan has fond memories of the Nexus 7 tablet, but that was nine years ago.

• 16:42David Pierce?

  And Google not only stopped making tablets, but stopped seeming to care about Android for tablets at all. Now both the hardware teams and the Android teams told me they're all in on both counts. So what gives here? Samir Samat, a VP of Product Management on the Android team, gave me what I think is pretty much the correct explanation.

• 17:00Speaker 02

  We were early pioneers in watches, for example. We were actually very early in multi-sized form factors for tablets. There's been a number of different form factors that we've worked on, but I think consumer expectations have changed as well over time. So the phone is certainly super important, but it's also becoming very clear that there are other device form factors which are complementary and also critical to a consumer deciding which ecosystem to buy into and which ecosystem to live in.

• 17:27David Pierce?

  That last thing he said, that thing about the ecosystem, that's the whole thing here. What Google wants is to not be a search engine, but to be a sort of AI companion that understands and adapts to you at all times. It's a cool vision, frankly, and it makes a lot of sense for Google to be following. It also requires Google to be essentially all things to all people at all times. It has to know everyone's taste, work on everyone's devices, speak everyone's language, understand everyone's workflows, and be perfectly personalized to every single person on planet Earth. That's a really high bar, even for Google. And if you're buying an iPhone because you love your Apple Watch, that makes life even harder for Google. Apple has long understood that an ecosystem is powerful, that you can sell devices and services

• 18:11David Pierce?

  through your other devices and services. Google, at long last, is finally figuring that out. That idea has forced Google to change a lot about how it works, even at a company level. Here's just one good example. For years, Google treated its Android and hardware teams like two totally separate things, and actually used to talk about how Android treated Pixel just like any other Android manufacturer. Now, Rick told me the two sides are getting much closer together.

• 18:38Speaker 04

  In our very earliest days, it was like, we're making platform software, and then we would make devices on that software. It's very different now in that we really sort of co-designed where things are headed. And I think that's just sort of the nature of how computers have changed, and computing models have changed. So, like, you know, the Better Together Initiative is like a key part of AMI computing. The reason we're doing both of these things is that this is our vision for how the model of computing changes in the future, and we're working on it together. We're making sure that our hardware complements the software that's being built and vice versa.

• 19:13David Pierce?

  That's one way this push towards ambient computing is changing Google, but it's not the only one. Google knows it's set up in this crazy, decentralized way, and it knows that to get to this ambient computing, everything-connected future, the company itself has to change to solve it. And if Google wants to do that, it's going to have to keep being focused on making all of its parts work together. Google has a long history of building something, getting sort of bored, and then building a new similar thing that doesn't work with the old thing. That's how you get 1,000 Google messaging apps over the years. That's why Google inexplicably operated two competing music services for a bunch of years. It's why it bought Nest, but also built its own operating system for the Internet of Things.

• 19:56David Pierce?

  It's why Google has worked on countless AR and VR projects, but has never actually had one that took off. The history of Google's failed projects is like a crazy alternate universe of the biggest stories in tech over the last 20 years. And it's rarely because Google is dead wrong, though sometimes it is, like when it tried to make Buzz and Wave and Google+ happen. It's because the culture of Google is to try everything and kill pretty much all of it. Why? Because frankly, Google doesn't need any of those things to work. It can continue to fritter away all those good ideas as long as the search ad money keeps chugging along. And make no mistake, it is still chugging along. But you've got to think that for Google, faced with antitrust reform and new privacy rules, and even companies like Amazon starting to take a bite of the ad business,

• 20:41David Pierce?

  Google's feeling a little more urgency as it thinks about what comes next. It also has to continue to convince users, many of whom are increasingly wary of how much Google knows about them, that the trade-off for all that data is worth it. And if you really cast out in the future, there's one other piece of this, too. If you think augmented reality is the future of computing, and a lot of people at Google do, this kind of ambient computer Google is building seems like exactly the kind of thing we're going to need. Nobody wants to walk down the street yelling, "Hey, Google, message Anna. I love you." We're also not going to want to switch between eight devices just to get things done. We're going to need computers that are massively versatile and switch seamlessly from place to place and task to task and input to input and don't particularly care

• 21:26David Pierce?

  about their form factor. Google wouldn't tell me much about what it's working on here for what it's worth. This is all I actually got out of Rick Osterloh.

• 21:33Speaker 04

  It's definitely a place where we are focusing as well. And we don't see the market as here now, but we definitely see it as a key part of the future.

• 21:42David Pierce?

  For the record, ambient computing also isn't here now. But Google has built a lot of the pieces to make it work without thinking much about how they work together. Now, the job for Google is to take all those pieces and turn them into something magical, something more than the sum of its parts. That comes in really small ways, like Google making it easier to move files or cast video between all your devices. And it comes in really big ways, like a virtual assistant you can talk to anywhere, about anything, any way you want. And that, at least it seems to me, is what Google is actually working on now, maybe for the first time. The ongoing question is, will this vision hold Google's interest long enough to get anywhere? Or will something shinier come along and just distract the company again? Who knows?

• 22:25David Pierce?

  Maybe next year's I O will be all about the metaverse and who cares about your life in the real world. But I would bet that if Google is ever going to find its next big thing after search ads, ambient computing might be the best chance it has. That, or really awesome AR glasses.

• 22:43David Pierce?

  That's it for the show today, but we are definitely not done talking about Google I O. We'll be back on Friday with all kinds of talk about all of the announcements from this week. Plus, we're going to be spending some time with Sundar Pichai, the CEO of Google and Alphabet, getting into all of the stuff about what Google's up to. So stick around. This special episode of The Vergecast was produced by me, David Pierce, Liam James, and Andrew Marino. The Vergecast is a production of the Vox Media Podcast Network. We'd love to hear from you about this show and everything. Please send us an email at [email protected]. Thanks for listening. We'll see you on Friday. Rock and roll.

• 23:48Speaker 08

  Trying to locate news sources that reliably separate fact from fiction can seem like looking for a needle in a haystack. That's why The Guardian is launching Stateside with Kai Wright and Carter Sherman, a conversation with experts who slow down the news and wrestle with the questions we all have about what's actually happening in the world. Three times a week, hosts Kai and Carter utilize all the reporting resources The Guardian has to discuss the news, international affairs, climate, culture, sports, lifestyle, fashion, and wellness. And The Guardian is not billionaire-owned, meaning they're free to report the whole picture without interference. Go to theguardian.com/stateside to learn more. And listen wherever you get your podcasts, or watch on YouTube. That's theguardian.com/stateside.