Large Action Models (LAMs) & Rabbits 🐇

Recently the release of the rabbit r1 device resulted in huge interest in both the device and “Large Action Models” (or LAMs). What is an LAM? Is this something new? Did these models come out of nowhere, or are they related to other things we are already using? Chris and Daniel dig into LAMs in this episode and discuss neuro-symbolic AI, AI tool usage, multimodal models, and more. Sponsors: Read Write Own https://readwriteown.com/?utm_source=changelog&utm_medium=practicalai&utm_campaign=changelog – Read, Write, Own: Building the Next Era of the Internet—a new book from entrepreneur and investor Chris Dixon—explores one possible solution to the internet’s authenticity problem: Blockchains. From AI that tracks its source material to generative programs that compensate—rather than cannibalize—creators. It’s a call to action for a more open, transparent, and democratic internet. One that opens the black box of AI, tracks the origins we see online, and much more. Order your copy of Read, Write, Own today at readwriteown.com https://readwriteown.com/?utm_source=changelog&utm_medium=practicalai&utm_campaign=changelogShopify https://www.shopify.com/practicalai – Sign up for a $1/month trial period at shopify.com/practicalai https://www.shopify.com/practicalaiFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: rabbit r1 https://www.rabbit.tech/Salesforce blog on LAMs https://blog.salesforceairesearch.com/large-action-models/LangChain tools https://python.langchain.com/docs/modules/agents/tools/MM-LLMs: Recent Advances in MultiModal Large Language Models https://huggingface.co/papers/2401.13601Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents. So you can launch your app near your users. Learn more at fly.io.

• 0:42Daniel Whitenack?

  Welcome to another fully connected episode of the Practical AI Podcast. In these fully connected episodes, we try to keep you up to date with everything that's happening in the AI and machine learning world and try to give you a few learning resources to level up your AI game. This is Daniel Whitenack. I'm the founder and CEO of Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who's a tech strategist at Lockheed Martin. How you doing, Chris?

• 1:12Chris Benson?

  Doing very well, Daniel. It's enjoying the day. And by the way, since you've traveled to the Atlanta area tonight, we haven't gotten together, but you're just a few minutes away, actually. So, welcome to Atlanta.

• 1:25Daniel Whitenack?

  Just got in. Yeah, we're within not maybe a short drive, depending on your view of what a short drive is.

• 1:34Chris Benson?

  Anything under three hours is short in Atlanta. And I think you're like 45 minutes away from me right now.

• 1:39Daniel Whitenack?

  Yeah, so hopefully we'll get a chance to catch up tomorrow, which will be awesome, because we rarely get to see each other in person. It's been an interesting couple of weeks for me. So for those that are listening from abroad, maybe, we had some major ice and snow type storms recently. And my great and embarrassing moment was I was walking back from the office and the like. Freezing rain and I slipped and fell and my laptop bag with laptop in it broke my fall, which is maybe good, but that also broke the laptop. So actually the laptop works.

• 2:24Daniel Whitenack?

  It's just the screen doesn't work. So maybe I'll be able to resolve that, but.

• 2:28Chris Benson?

  It's like a mini portable server there, isn't it?

• 2:30Daniel Whitenack?

  Yeah, exactly. You know, you have enough monitors around. It's not that much of an issue. But yeah, I had to put Ubuntu on a burner laptop for the trip. So yeah,

• 2:42Chris Benson?

  Okay.

• 2:43Daniel Whitenack?

  It's always a fun time. Speaking of personal devices, there's been a lot of interesting news and releases, not of, well, I guess of models, but also of interesting actual hardware devices related to AI. Recently, one of those is the Rabbit R1, which was announced and sort of launched to pre-orders with a lot of acclaim. Another one that I saw was the AI pen, P I N, which is like a little, I don't know, my grandma would call it a brooch, maybe thing, like a large pen you put on your jacket

• 3:26Daniel Whitenack?

  or something like that. I am wondering, Chris, as you see these devices, and I want to dig a lot more into some of the interesting research and models and data behind some of these things like Rabbit. But just generally, what are your thoughts on this sort of trend of AI-driven personal devices to help you with all of your personal things and plugged into all of your personal data and sort of AI attached to everything in your life?

• 3:57Chris Benson?

  Well, I think it's coming. Maybe it's here, but I know that I am definitely torn. I mean, I love the idea of all this help along the way. There's so many, like I forget everything. I'm terrible if I don't write something down and then follow up on the list. I am not a naturally organized person. So my wife is, and my wife is always reminding me that I really struggle in this area. And usually she's, she's not being very nice in the way that she does it. So it's all love, I'm sure. But

• 4:26Daniel Whitenack?

  Yes, yes.

• 4:27Chris Benson?

  So part of me is like, wow, this is the way I could actually. You know, be all there, get all the things done. But the idea of just giving up all my data and just being, uh, it's like so many others. It's not, uh, that aspect is not appealing. So yeah, I guess I'm, I, I'm not leaping.

• 4:47Daniel Whitenack?

  How much different do you think this sort of thing is than everything we already give over with our smartphones?

• 4:55Chris Benson?

  It's a good point you're making.

• 4:56Daniel Whitenack?

  I mean, we've had computing devices with us in our pocket or on our person 24/7 for what at least the past 10 years, you know, for at least for those that adopted the iPhone or whatever when it came out. But, um, yeah, I, so in terms of location, certainly account access and certain automations. What do you think makes it? Because obviously this is something on the mind of the makers of these devices. Cause I think both the AI pen and the rabbit R1 make some sort of explicit statements in their launch and in their website about privacy is really important to us.

• 5:39Daniel Whitenack?

  This is how we're doing things to, because we really care about this. So obviously they anticipated some kind of additional reaction, but we, we all already have smartphones. I think most of us, if we are willing to admit it, we know that we're being tracked everywhere and all of our data goes everywhere. So I don't know, what is it about this AI element that you think either makes a, an actual difference in terms of the substance of what's happening with the data? Or is it just a perception thing?

• 6:09Chris Benson?

  It's probably a perception thing with me. I mean, because everything that you said, I agree with you. You're dead on. And we've been giving this data out for years and we've gotten comfortable with it. And that's just something that we all kind of don't like about it, but we've been accepting it for years. And I guess it's the expectation that with these AI assistants that we've been hearing about for so long coming. And we're starting to see things like the rabbit come into market and such, that there's probably a whole new level of kind of analysis of us and all the things. And in a sense, knowing you better than you do, that is uncomfortable and probably will not be as uncomfortable in the years to come, because we'll grow used to that as well.

• 6:53Chris Benson?

  But I have to admit right now, it, it's an emotional reaction. It makes me a little bit leery.

• 6:57Daniel Whitenack?

  Yeah, maybe it's prior to these sorts of devices. There was sort of the perception at least that yes, my data is going somewhere. Maybe there's an nefarious person behind this, but there's sort of a person behind this. Like the data is going all to Facebook or Meta. And they're like, maybe they're even listening in on me and putting ads for mattresses in my feed or whatever the thing is, right? So that perception has been around for quite some time, regardless of whether Facebook is actually listening in or whatever, or it's another party like, you know, the NSA and the government's listening in.

• 7:41Daniel Whitenack?

  But I think all of those perceptions really relied on this idea that even if there's something bad happening that I don't want happening with my data, there's sort of a group of people back there doing something with it. And now there's this sort of idea of this agentic entity behind the scenes that's doing something with my data without human oversight. I think maybe that's, if there's anything sort of fundamentally different here, I think it's the level of automation and the sort of agentic nature of this, which does provide some sort of difference. Although there's always like, you know, if you're processing voice or something, there's voice analytics and you can put that to text.

• 8:26Daniel Whitenack?

  And then there are always NLP models in the background doing various things or whatever. So there's some level of automation that's already been there, but.

• 8:34Chris Benson?

  I agree. And I think, but you mentioned perception upfront, and I think that makes a big difference. I guess with, like you mentioned NSA, intelligence agencies, I think we all just assume that they're all listening to all the things all the time now. And that's one of those things that's completely beyond your control. And so there's almost no reason to worry about it, I suppose, unless you happen to be one of the people that an intelligence agency would care about, which I don't particularly think I am.

• 9:02Daniel Whitenack?

  Yeah.

• 9:03Chris Benson?

  So it just goes someplace and you just kind of shrug it off. There's a certain amount of what we've done these years with mobile where it's, you're opting in. I think it's leveling up to, we're saying with some of these AI agents coming out, we know how much data about ourselves is going to be there. And so it's just escalating the opt-in up to a whole new level. So hopefully we'll see what happens.

• 9:25Daniel Whitenack?

  Yeah.

• 9:25Chris Benson?

  Hope it works out well.

• 9:26Daniel Whitenack?

  We haven't really, for the listeners, maybe that are just listening to this and haven't actually, maybe you're in parallel doing the search and looking at these devices. But in case you're on your run or in your car, we can describe a little bit. So I described the AI pen thing a little bit. The Rabbit, I thought, was really, really cool design. I don't know if there's any nerds out there that love this sort of synthesizer analog sequencer, teenage engineering stuff that's out there. But actually, the sort of hardware design, teenage engineering was involved in that in some way. So it's like a little square thing, the Rabbit R1.

• 10:08Daniel Whitenack?

  It's got like one button you can push and speak a command. It's got a little hard, actual hardware wheel that you can spend to scroll. And the screen is kind of just, they show it as black most of the time, but it pops up with, you know, the song you're playing on Spotify or some of the things you would expect to be happening on a touchscreen or that sort of thing. But the primary interface is thought to be, in my understanding, speech. Not that you would be pulling up a keyboard on the thing and typing in a lot. That's kind of not the point. The point would be this sort of speech-driven conversational, and they even call it an operating system, conversational operating system

• 10:53Daniel Whitenack?

  to do certain actions or tasks, which we'll talk a lot more about the kind of research behind that. But that's kind of what the device is and looks like.

• 11:03Chris Benson?

  It's interesting that, you know, going with the device route, you know, and the fact that they're selling the actual unit itself. And, you know, over the years, we've been, you know, we started on our computer, or well, we started on desktops and then went to laptops and then went to our phones. And the phones have evolved over time. And we've been talking about, you know, wearables and things like that over the years as they've evolved. But I think there's a little bit of a gamble in actually having it as a physical device, because that's something else that they're presuming you're going to put at the center of your life. That versus being kind of the traditional phone app approach, where you're using the thing that they are, that your customer already has in their hands. What are your thoughts about the physicalness of this offering?

• 11:46Daniel Whitenack?

  I think it's interesting, one of the points, if you watch the release or launch or promotion video for the Rabbit R1, he talks about sort of the app-driven nature of smartphones and there's an app for everything. And there's so many apps now that navigating apps is kind of a task in and of itself. And the Silicon Valley meme, no one ever deletes an app, right? So you just accumulate more and more and more apps and they kind of build up on your phone. And now you have to organize them into little groupings or whatever. So I think the point being that it's nice that there's an app for everything,

• 12:27Daniel Whitenack?

  but the navigation and orchestration of those various apps is sometimes not seamless and burdensome. I'm even thinking about myself and kind of checking over here, you know, I got in the Uber. Oh, I forgot to switch over my payment on my Uber app. So now I've got to open my bank app, right? And then grab my virtual card number and copy that over. But then I've got to go to my password management app to like copy my password. Or like, there's all these sorts of interactions between various things that aren't seamless as you might think they would be. But it's easy for me to say in words, conversationally, hey, I want to update

• 13:11Daniel Whitenack?

  the payment on my current Uber ride or whatever, right? So the thought that that would be an easy thing to express conversationally is interesting. And then have that be accomplished in the background if it actually works is also quite interesting.

• 13:28Chris Benson?

  I agree with that. And I can't help but wonder, if you look back at the advent of the phone and the smartphone and you know, the iPhone comes out and it really isn't really so much a phone anymore, but a little computer. And so we kind of, the idea of the phone being the base device in your life has been something that's been with us now for, you know, over 15 years. And so one of the things I wonder is, could there be a trend where maybe the phone doesn't become... If you think about it, you're texting, but a lot of your texting isn't really texting. It's messaging and apps. Maybe the phone is no longer the central device in your life going forward. And maybe you're actually having your primary thing. And so that would obviously play into Rabbit's approach where they're giving you another device.

• 14:13Chris Benson?

  It packages everything together in that AI OS that they're talking about. Or conversationally, it runs your life. If you expose your life to it, the way you are across many apps on the phone. But it's an opportunity potentially to take a left turn with the way we think about devices. And maybe the phone is no longer in the future, in the not so distant future. Maybe the phone is no longer the centerpiece.

• 14:49Speaker 03

  If you're listening, you know that artificial intelligence is revolutionizing the way we produce information, changing society, culture, politics, the economy. But it's also created a world of AI-generated content, including deep fakes. So how can we tell what's real online? Read, Write, Own: Building the Next Era of the Internet. A new book from entrepreneur and investor Chris Dixon explores one possible solution to the internet's authenticity problem: blockchains. From AI that tracks its source material to generative programs that compensate rather than cannibalize creators. Read, Write, Own is a call to action for a more open, transparent, and democratic internet.

• 15:32Speaker 03

  One that opens the black box of AI, tracks the origins we see online, and much more. This is our chance to reimagine world-changing technologies to build the internet we want, not the one we inherited. Order your copy of Read, Write, Own today or go to ReadWriteOwn.com to learn more.

• 16:13Daniel Whitenack?

  All right, Chris. Well, there's a few things interacting in the background here in terms of the technology behind the Rabbit device and I'm sure other similar types of devices that have come out. Actually, there's some of this sort of technology that we've talked a little bit about on the podcast before. I don't know if you remember, we had the episode with Ask UI, which they had this sort of multimodal model that I think a lot of their focus over time was on testing. You know, a lot of people might test web applications or websites using something like Selenium or something like that that automates desktop

• 16:55Daniel Whitenack?

  activity or interactions with web applications and actually automates that for testing purposes or other purposes. And Ask UI had some of this technology a while back to kind of perform certain actions using AI on a user interface without sort of hard coding, like click on 100 pixels this way and 20 pixels down this way, right? So that I think has been going on for some time. This adds a sort of different element to it in that there's the voice interaction, but then they're really emphasizing the flexibility of this and the updating of it. So actually, they emphasize like, I think

• 17:38Daniel Whitenack?

  some of the examples they gave is, you know, I have a certain configuration on my laptop or on my screen that I'm using with a browser with certain plugins that make it look a certain way and everything sort of looks different for everybody and it's all configured in their own sort of way. Even app wise, apps kind of are very personalized now, right? Which makes it a challenge to say, click on this button at this place. It might not be at the same place for everybody all the time. And of course, apps update and that sort of thing. So the solution That Rabbit has come out with to deal with this is what they're calling a large action model.

• 18:21Daniel Whitenack?

  And specifically, they're talking about this large action model being a neuro symbolic model. And I want to talk through a little bit of that, but before I do, I think we sort of have to back up and talk a little bit about. AI models, large language models, ChatGPT has been interacting with external things for some time now. I think there's confusion at least about how that happens and you know what the model is doing. So it might be good just to kind of set the stage for this. In terms of how these models are interacting with external things. So the way that this looks, at least in the Rabbit case is you click the button and you say.

• 19:05Daniel Whitenack?

  Oh, I want to change the payment card on my Uber unclick and like stuff happens in the background. And somehow the large action model interacts with Uber and maybe my bank app or whatever. And actually makes the update, right? So the question is how this happens. Have you used any of the plugins or anything in ChatGPT or the kind of search the web? Type of plugin to a chat interface or anything like that?

• 19:34Chris Benson?

  Absolutely. I mean, that's what makes the, I mean, I think people tend to focus on the model itself, you know? I mean, that's where all the glory is and people say, ah, this model versus that. But so much of the power comes in the plugins themselves and or other ways in which they interact with the world. And so as we're trying to kind of pave our way into the future and figure out how we're going to use these. And how they're going to impact our lives, whether it be the Rabbit way or whether you're talking ChatGPT with its plugins. That's the key. It's all those interactions is the touch points with the different things that you care about, which makes it worthwhile. So yes, absolutely. And I'm looking forward to doing it some more here.

• 20:14Daniel Whitenack?

  Yeah, so there's a couple of things maybe that we can talk about and actually, some of them are even highlighted in recent things that happen that we may want to highlight also. One of those is if you think about a large language model like that used in ChatGPT or, you know, neural chat, Llama 2, whatever it is, you put text in and you get text out. We've talked about that a lot on the show. And so you put your prompt in and you get a completion. It's like fancy autocomplete and you get this completion out, right? Not that interesting. We've talked a little bit about RAG on the show, which means I am programming

• 20:55Daniel Whitenack?

  some logic around my prompt such that when I get my user input, I'm searching some of my own data or some external data that I've stored in a vector database or in a set of embeddings to retrieve text that's semantically similar to my query and just pushing that into the prompt as a sort of grounding mechanism to sort of ground the answer in that external data. So you've got sort of basic autocomplete. You've got retrieval to insert external data via a vector database. You've got some multimodal inputs. So and by multimodal models, I'm meaning things like Lava.

• 21:40Daniel Whitenack?

  And actually, this week, there was a great published on January 24th. I saw it in the daily papers on Hugging Face. MMLM's recent advances in multimodal large language models. So if you're wanting to know sort of the state of the art and what's going on in multimodal large language models, like I just mentioned, that's probably a much deeper dive that you can go into. So check out that and we'll link it in our show notes. But these are models that would not only take a text prompt, but might take a text prompt paired with an image, right? So you could put an image in and you say, Also have a text prompt that says, is there a raccoon in this image?

• 22:22Daniel Whitenack?

  Right? And, and you know, hopefully the reasoning happens and says yes or no. If there's a Isn't there always a raccoon in the image? There's, there's always a raccoon everywhere. That's one element of this as a, that would be a specialized model that allows you to integrate multiple modes of data. And there's similar ones out there for audio and text and other things. So again, summary, you've got text to text autocomplete. You've got this retrieval mechanism to pull in some external text data into your text prompt. You've got specialized models that allow you to bring in an image and text.

• 23:04Daniel Whitenack?

  All of that's super interesting and I think it's connected to what Rabbit is doing. But there's actually more to what's going on with, let's say, when people perform actions on external systems or integrate external systems with these sorts of AI models. And this is what in the sort of Langchain world, if you've interacted with Langchain at all, they would call this maybe tools. And you even saw things in the past, like Toolformer and other models, where the idea was, well, okay, I have, uh, maybe it's the Google Search API, right? Or SERP API or one of these search APIs, right?

• 23:48Daniel Whitenack?

  I know that I can take a JSON object, send it off to that API and get a search result, right? Okay, so now if I want to call that search API with an AI model, what I need to do is get the AI model to generate the right JSON structured output that I can then just programmatically, not with any sort of fancy AI logic, but programmatically take that JSON object. And send it off to the API, get the response and either plug that in in the sort of retrieval way that we talked about before, or just give it back to the user as the response that they wanted, right? So this has been happening for quite a while.

• 24:32Daniel Whitenack?

  This is kind of like, we saw one of these cool AI demos every week, right? Where, oh, the AI is integrated with Kayak now to get me a rental car and the AI is integrated with, you know, this external system and all really cool, but at the heart of that was the idea that I would generate structured output that I could use in a regular computer programming way to call an API and then get a result back, which I would then use in my system. So that's, that's kind of this tool idea, which is still not quite what Rabbit is doing, but I think that's something that people don't realize is happening behind the scenes in these tools.

• 25:15Chris Benson?

  I think that's really popular in the enterprise, you know, and I, you know, in the enterprise with air quotes there, because that approach is, you know, in large organizations, they're going to other, you know, the, the cloud providers with their APIs, you know, Microsoft is, you know, has the relationship with OpenAI and they're wrapping that, you know, Google has their APIs and they're using RAG, you know, in that same way to try to integrate with systems instead of actually creating the models on their own. I would say that's a very, very popular approach right now in enterprisey environments that are still more software driven and still trying to figure out how to use APIs for AI models.

• 25:55Daniel Whitenack?

  Yeah, and I can give you a concrete example of something we did with a customer Prediction Guard, which is the Shopify API, right? So e-commerce customer, the Shopify API has this sort of Shopify, I think it's called Shopify QL, Query Language. It's structured, right? And you can call the regular API via GraphQL, right? And so it's a very structured sort of way you can call this API to get sales information or Order information or do certain tasks, right? And so you can create a natural language query and say, "Okay, well, don't try to give me natural language out,

• 26:37Daniel Whitenack?

  but give me Shopify QL or give me something that I can plug into a GraphQL query and then I'm gonna go off and query the Shopify API and either perform some interaction or get some data, right? So this is very popular. This is how you sort of get AI on top of tools. What's interesting, I think, that Rabbit observes in what they're saying and others have observed as well. I think, you know, you take the case like AskUI, like we talked about before. And the observation is that not everything has this sort of nice structured way you can interact with it with an API. So think about, pull out your phone. You've got all of these apps on your phone.

• 27:23Daniel Whitenack?

  Some of them will have a nice API that's well-defined. Some of them will have an API that, me as a user, I know nothing about, right? There's maybe an API that exists there, but it's hard to use or not that well documented, or maybe I don't have the right account to use it or something. There's all of these interactions that I wanna do on my accounts with my web apps, with my apps, that have no defined structured API to execute all of those things. So then the question comes, and that's why I wanted to lead up to this, is because even if you can retrieve data to get grounded answers,

• 28:05Daniel Whitenack?

  even if you can integrate images, even if you can interact with APIs, all of that gets you pretty far, as we've seen. But ultimately, not everything is gonna have a nice structured API or it's not gonna have an API that's updated or has all the features that you want or does all the things you want, right? So the question, I think the fundamental question that the Rabbit research team is thinking about is, how do we then reformulate the problem in a flexible way?

• 28:37Daniel Whitenack?

  To allow a user to trigger an AI system to perform arbitrary actions across an arbitrary number of applications or an application without knowing beforehand the structure of that application or its API. So I think that's the really interesting question.

• 28:57Chris Benson?

  I agree with you completely. And you know, there's so much complexity that they refer to it as human intentions expressed through actions on a computer. And that sounds really, really simple, you know, when you say it like that, but there's so, that's quite a challenge to make that work in an unstructured world. So I'm really curious. I haven't, they have their research page, but I don't guess they've, they've put out any papers that describe some of the research they've done yet, have they?

• 29:27Daniel Whitenack?

  Just in general terms, and that's where we get to the, the exciting world of large action models.

• 29:38Chris Benson?

  Somehow that makes me think of like Arnold Schwarzenegger.

• 29:41Daniel Whitenack?

  Large, large action heroes.

• 29:44Chris Benson?

  There you go. Exactly.

• 30:07Speaker 03

  You know, when we started podcasting back in 2009, and online store is just the furthest thing from our minds. Now we have merch.changelog.com and you can go there right now and order some t-shirts and that's all powered by Shopify. It's so easy, all because Shopify is amazing. Shopify is the global commerce platform that helps you sell at every stage of your business. From the launch your online shop stage to the first real life store stage, all the way to the, did we just hit a million dollar stage? Shopify is there to help you grow. Whether you're selling security systems or marketing memory modules, Shopify helps you sell everywhere from their All in one ecommerce platform to their in-person POS system,

• 30:50Speaker 03

  wherever and whatever you're selling, Shopify has got you covered. Shopify helps you turn browsers into buyers with the internet's best converting checkout, up to 36% better compared to other leading commerce platforms, and sell more with less effort thanks to Shopify Magic, your AI-powered all-star. You know, nothing gets me and Jared more excited than when our guests get that coupon code in their email when their show ships, or to everyone out there who loves Change Law Podcasts and can go to merch.changelog.com and get your favorite threads to support our podcasts. It is just the best thing ever. From stickers to threads, all that is at merch.changelog.com. And did you know that Shopify powers 10% of all ecommerce in the US?

• 31:33Speaker 03

  And Shopify is the global force behind Allbirds, Rothy's, and Brooklinen, and millions of other entrepreneurs of every size across 175 countries. Plus, Shopify's extensive help resources are there to support you and your success every step of the way, because businesses that grow, grow with Shopify. Sign up for a $1 per month trial period at Shopify.com/PracticalAI. All lowercase, go to Shopify.com/PracticalAI now to grow your business no matter what stage you're in. Again, Shopify.com/PracticalAI.

• 32:34Daniel Whitenack?

  Yeah, Chris, so coming from Arnold Schwarzenegger and large action heroes to large action models, I was wondering if this was a term that Rabbit came up with? I think it has existed for some amount of time. I at least saw it at least as far as back as June of last year, 2023, I saw. Silvio Savaris article on Salesforce AI research blog about LAMs from large language models to large action models. I think the focus of that article was very much on the sort of agentic stuff that we talked about before in terms of interacting with different systems, but in a very automated

• 33:19Daniel Whitenack?

  way. The term large action model, as far as Rabbit refers to it, it's this new architecture that they are saying that they've come up with, and I'm sure they have because it seems like the device works. We don't know, I think, all of the details about it. At least I haven't seen all of the details, or it is sort of not transparent in the way that maybe a model release would be on hugging face with code associated with it in a long research paper. Maybe I'm missing that somewhere, or listeners can tell me if they've found it. But I couldn't find that. They do have a research page, though, which gives us a few clues as to what's going on

• 34:02Daniel Whitenack?

  and some explanation in kind of general terms. And what they've described is that their goal is to observe human interactions with a UI. And there seems to be some sort of multimodal model that is detecting what things are where in the UI. And they're mapping that on to some kind of flexible, symbolic, synthesized representation of a program. So the user is doing this thing, right? So I'm changing the payment on my Uber app, and that's represented or synthesized behind the scenes.

• 34:48Daniel Whitenack?

  In some sort of structured way and kind of updated over time as it sees demonstrations, human demonstrations of this going on. And so the words that they, I'll just kind of read this so people, if they're not looking at the article, they say, "We designed the technical stack from the ground up, from the data collection platform to the new network architecture." And here's the sort of very dense loaded wording that probably has a lot packed into it. They say that utilizes both transformer-style attention and graph-based message passing,

• 35:27Daniel Whitenack?

  combined with program synthesizers that are demonstration and example-guided. So that's a lot in that statement, and of course, they mention a few in more description and in other places. But it seems like my sort of interpretation of this is that the requested action comes in to the system, to the network architecture, right? And there's a neural layer. So this is a neural symbolic model. So there's a neural layer that somehow interprets that user action into a set of symbols or

• 36:09Daniel Whitenack?

  representations that it's learned about the AI, like the UI, I mean, the Shopify UI or the Uber UI or whatever. And then they use some sort of symbolic logic processing of this sort of synthesized program to actually execute a series of actions within the app and perform an action that it's learned through demonstration. So this is sort of what they mean, I think, when they're talking about neuro-symbolic. So there's a neural network portion of this, kind of like when you put something into ChatGPT or a transformer-based large language model and you get something out.

• 36:53Daniel Whitenack?

  In the case of, we were talking about getting JSON structured out. When we're interacting with an external tool, but here it seems like you're getting some sort of thing out, whatever that is, a set of symbols or some sort of structured thing. That's then passed through symbolic processing layers that are essentially symbolic and rule-based ways to execute a learned program over this application. And by program here, I think they mean, they reference a couple of papers and my best interpretation is that they mean not a computer program in the sense of Python code, but a logical program

• 37:38Daniel Whitenack?

  that represents an action like here is the logical program to update the payment on the Uber app. You go here and then you click this and then you enter that and then you blah, blah, you do those things, right? Except here, those programs, those synthesized programs are learned by looking at human intentions and what they do in an application. And that's how those programs are synthesized. So that was a long, I don't know how well that held together, but that was my my best at this point without seeing anything else from a single sort of blog post.

• 38:19Chris Benson?

  When you can keep me quiet for a couple of minutes there, it means you're doing a pretty good job. I have a question I want to throw out and you won't, I don't know that you'll, you'd be able to answer it obviously, but it just to speculate. While we were talking about that and thinking about multimodal, I'm wondering the device itself comes with many of the same, you know, sensors that you're going to find in a cell phone these days. But I'm wondering if that feeds in more than just the, the speech and that obviously has the camera on it. It comes with a magnetometer, I can't say the word, GPS, accelerometer and gyroscope. And obviously, so you know, it's detecting motion, it knows location, all the things, has the camera, has the mic.

• 39:03Chris Benson?

  How much of that do you think is relevant to the lambs, to the large action model in terms of inputs? Do you think that there is potentially relevance in the non-speech and non-camera concerns on it? Do you think the way people move could have some play in there? I know we're being purely speculative. I'm just, it caught my imagination.

• 39:25Daniel Whitenack?

  Yeah, I'm not sure. I mean, it could be that that's used in ways similar to how those sensors are used on smartphones these days, right? Like if I'm asking Rabbit to book me an Uber to here or something like that, right? Now, it could infer the location maybe of where I am based on where I'm wanting to go, or ask me where I am. But likely, the easiest thing would be to use a GPS sensor, right, to know my location and just like put that as the PIN in the Uber app. And now it knows. So I think there's some level of interaction

• 40:06Daniel Whitenack?

  between these things. I'm not sure how much, but it seems like, at least in terms of location, I could definitely see that coming into play. I'm not sure on the other ones.

• 40:16Chris Benson?

  Well, it looks a lot like a, physically, it looks like a lot like a smartphone without the phone.

• 40:21Daniel Whitenack?

  Yeah, yeah, a smartphone, different sort of aspect ratio, but still kind of touchscreen. I think you can still pull up a keyboard and that sort of thing. And you know, you see things when you prompt it. So yeah, I imagine that that's maybe an evolution of this over time is sensory input of various things. Like I could imagine that being very interesting in running or fitness type of scenarios, right? If I've got my Rabbit with me and I instruct Rabbit to post a celebratory social media post every time I keep my mileage

• 41:04Daniel Whitenack?

  or my time per mile at a certain level or something, and it's using some sort of sensors on the device to do that. I think there's probably ways that that will work out. I'm not sure about now.

• 41:17Chris Benson?

  It'll be interesting that if this approach sticks, and I might make an analogy to things like the Oura Ring, you know, for health, wearing that. And then competitors started coming out. And then, you know, Amazon has their own version of a, of a health ring that's coming out. Along those lines, you have all these incumbent players in the AI space that are, you know, for the most part, very large, well-funded, you know, cloud companies, and in, in at least one case, a retail company blended in there. And so if this might be an alternative in some ways to the smartphone being the dominant

• 41:58Chris Benson?

  device, and it has all the same capabilities plus more, and they have the lamb behind it to drive that functionality, how long does it take for an Amazon, or a Google, or a Microsoft to come along after this and start producing their own variant? Because they already have the infrastructure that they need to produce the backend, and they're going to be able to produce, you know, Google and Amazon certainly produce front-end stuff quite a lot as well. So it'll be interesting to see if this is the beginning of a new marketplace opening up in the AI space as an entrant.

• 42:32Daniel Whitenack?

  So there's already really great hardware out there for smartphones, and I wonder if something like this is kind of a shock to the market. But in some ways, you know, just as phones with external key buttons sort of morphed into smartphones with touchscreens, otherwise I could see smartphones that are primarily app-driven in the way that we interact with them now, being pushed in a certain direction because of these interfaces. And so smartphones won't look the same in two years as they do now, and they won't

• 43:15Daniel Whitenack?

  follow that same sort of app-driven trajectory like they are now, probably because of things that are rethought. And it might not be that we all have rabbits in our pocket, but maybe smartphones become more like rabbits over time. I'm not sure. I think that that's very likely a, a thing that happened. It, it's also interesting to me, it's a little bit hard to parse out for me what's happening. What's the workload like between what's happening on the device? And what's happening in the cloud? And what sort of connectivity is actually needed for full functionality with the device?

• 43:57Daniel Whitenack?

  Maybe that's something if you want to share your own findings on that in our Slack community at changelog.com/community, we'd love to hear about it. My understanding is there is at least a good portion of the lamb and the lamb powered routines that are operating in a centralized sort of platform and hardware. So there's not this kind of huge, large model running on a very low power device that might suck away all the, all the energy. But I think that's also an interesting direction is how far could we get, especially with local models getting so good recently with fine tuned, local optimized, quantized models

• 44:41Daniel Whitenack?

  doing action related things on edge devices in our pockets that aren't relying on stable and high speed internet connections, which also of course helps with the privacy related issues as well.

• 44:59Chris Benson?

  I agree. By the way, I'm going to make a prediction. I'm predicting that a large cloud computing service provider will purchase Rabbit.

• 45:09Daniel Whitenack?

  All right. You heard it here first. I don't know what sort of odds Chris is, is giving, or I'm not going to bet against him, that's for sure. But yeah, I think, I think that's interesting. There's definitely a lot of, I think there will be a lot of action models of some type, whether those be tool using LLMs or LAMs or SLMs or whatever, whatever we've got coming up.

• 45:39Chris Benson?

  And they should have named it, they could have named it a lamb instead of a rabbit. You know, I just want to point out they're, they're getting their animals mixed up, man.

• 45:46Daniel Whitenack?

  I actually, yeah, that's a really good point. I don't know if they came up with rabbit before lamb, but maybe they just had the lack of the B there. But I think they probably could have figured out something.

• 45:58Chris Benson?

  Yeah. And, and the only thing that could have been the, just in the lamb is raccoon, of course, but that's beside the point. You have to come around, full circle there.

• 46:06Daniel Whitenack?

  Of course, of course. We'll, we'll leave that device up to you as well. Yeah. All right. Well, this has been fun, Chris. I do recommend in terms of if people want to learn more, there's a really good research page on rabbit.tech, rabbit.tech/research. And down at the bottom of the page, there's a list of references that they share throughout that people might find interesting as they explore the technology. I would also recommend that people look at Langchain's documentation on tools. And also, maybe just check out a couple of these tools. They're not that complicated. Like I say, there's sort of,

• 46:49Daniel Whitenack?

  they expect JSON input and then they run a software function and do a thing. That's sort of what's happening there. So maybe check out some of those in the array of tools that people have built for Langchain and try using them. So yeah, this has been fun, Chris.

• 47:04Chris Benson?

  Thanks. It was great. Thanks for bringing the rabbit to our attention.

• 47:07Daniel Whitenack?

  Yeah. Hopefully see you in person soon.

• 47:10Chris Benson?

  That's right.

• 47:11Daniel Whitenack?

  And yeah, we'll include some links in our show notes. Everyone take a look at them. Talk to you soon, Chris.

• 47:17Chris Benson?

  Have a good one.

• 47:27Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freakin' residents, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.