There's a new Llama in town

It was an amazing week in AI news. Among other things, there is a new NeRF and a new Llama in town!!! Zip-NeRF can create some amazing 3D scenes based on 2D images, and Llama 2 from Meta promises to change the LLM landscape. Chris and Daniel dive into these and they compare some of the recently released OpenAI functionality to Anthropic’s Claude 2. Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Typesense https://cloud.typesense.org/?utm_source=changelog – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You literally can’t get any faster! Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: What is NeRF article https://datagen.tech/guides/synthetic-data/neural-radiance-field-nerf/Llama 2: Llama 2 site https://ai.meta.com/llama/Llama 2 paper https://arxiv.org/pdf/2302.13971.pdfOpenAI Code Interpreter https://openai.com/blog/chatgpt-plugins#code-interpreterAnthropic Claude 2 https://www.anthropic.com/index/claude-2Learning resources: Hugging Face guide to Llama 2 https://huggingface.co/blog/llama2LLaMA 2 - Every Resource you need https://www.philschmid.de/llama-2OpenAI code interpreter article https://www.oneusefulthing.org/p/what-ai-can-do-with-a-toolbox-gettingUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users, no ops required. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another Fully Connected episode of Practical AI. In these episodes, Chris and I keep you fully connected with everything that's happening in the AI community. We're going to take some time to discuss the latest AI news and then we'll share some learning resources to help you level up your machine learning game. This is Daniel Whitenack. I'm a founder and data scientist at Prediction Guard and I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 1:16Chris Benson?

  Doing cool. I'm trying to figure out how did we survive before all these great new models and stuff. Like, it's changed my...

• 1:24Daniel Whitenack?

  Yeah, it's been crazy. Yeah, I just created a post for LinkedIn and I was like, grabbing text, putting it into ChatGPT, like getting nice rephrasing and then like, "Oh, I need an image." And in particular, we'll talk about it a little bit in this episode, but I was like, "Oh, there's this Free Willy model from Stable AI, which is like whale-themed, right?" And then I've got the llama thing. So I just went to Stable Diffusion Excel on ClipDrop and said, "Hey, generate me an image with a whale and a llama together." And, you know, how did I even post to LinkedIn before without these things? It's like a different world.

• 2:07Chris Benson?

  Yeah. 2023 versus 2022 is totally different.

• 2:11Daniel Whitenack?

  The content generation, the way you code, it's a different world. Yeah. And this week, as most weeks are, it seems like in 2023, had some pretty groundbreaking announcements and releases, which we're going to dive into a bunch of those things. There's just a huge amount to update on. And I think it's a good time for one of these episodes between you and I to just parse through some of the new stuff that is hitting our feeds. So, yeah. Well, I mentioned Llama. One of the big things this week was Llama 2, but I think before we jump into Llama 2, which I think was maybe the main thing dominating at least my world this

• 2:56Daniel Whitenack?

  week, it might be worth just taking a little bit of time to highlight something outside of this stream of large language models, which also crossed my desk this week, which I thought was really cool. It's this latest version of NERF. This is work from Google, presented at ICCV 2023. So it's called Zip NERF: Anti-Alias Grid-Based Neural Radiance Field. That's quite a name right there. It is quite a name. It stands for Neural Radiance Field. So NERF, it's like camel-cased

• 3:34Daniel Whitenack?

  in capital N, small e, and capital RF. These are fully-connected neural networks that create unique, novel views of complicated 3D scenes based on a set of images that are input. So I don't know if you've seen that video yet. I'm looking at it as we are talking. And when you say the video, I know which video you're talking about because it's amazing. I've just left it on. It's pretty spectacular. Yeah. So this is a podcast, so it's hard to express some of this for people. If you just search for Zip NERF, you can go to the page for this paper,

• 4:14Daniel Whitenack?

  which is a great summary. But there's a video on the page. And just to describe what it is, imagine- This kind of complicated house with a bunch of different rooms and an outdoor patio sort of garden area. And the video is actually this kind of almost like a drone fly through of the house and then the outdoor area. If you imagine a drone flying through a house, there's hats and coats and toys and couches and plants and all sorts of things everywhere. But the video is extremely seamless and it's not generated by a drone. It's actually just generated by interpolating between a whole bunch of 2D images.

• 5:00Daniel Whitenack?

  And then interpolating from that, the 3D scene. So yeah, I don't know. What are your impressions, Chris?

• 5:07Chris Benson?

  First of all, the, from the perspective, the drone flight, if you will, that you have as a perspective viewing it is like the best drone operator in the history of the world.

• 5:16Daniel Whitenack?

  Yeah, there'd probably be hard to get one to do that. Yeah.

• 5:18Chris Benson?

  Yeah, you're not going to get a real drone operator that could fly that amazingly, you know, and get those things. It's just phenomenal. And the house is like, for a moment you look at it and I mean, it is, it looks real. But I have noticed it's cluttery, but it's immaculately clean at the same time as well. The clutter is cleanly distributed and stuff. So I wish when my house was cluttered, it looked as beautiful as this house. It doesn't. But, but yeah, I mean, just like, if you didn't know, if you weren't listening to the, you know, Practical AI podcast to go look at it or something like that, and you just stumbled upon it.

• 5:58Chris Benson?

  You'd think it was a drone video, you know, if you didn't have the education and go, oh my God, this is just really cool. I wonder how, you know, wonder what they're doing here. But it's indistinguishable from real life for all practical purposes.

• 6:09Daniel Whitenack?

  Yeah. And so it's based on 2D images and then there are these generated interpolations, which maybe gets to, there was something that we were talking about prior to hitting the record button, which was this whole field of generative AI is sometimes conflated with large language models or chat GPT. But there's a whole lot going on in generative AI that's not language related or. Maybe even based on language-related prompts. So I mentioned that image that I generated for my LinkedIn post, that was still a text prompt into a model that generated an image.

• 6:49Daniel Whitenack?

  But here, what we're seeing is we've got static 2D images that are input to a model that's actually generating a whole bunch of different perspectives that are synthesized in a 3D scene. So this is, I would say, still fitting into our current landscape and world of generative AI, but it's not a text in, text out, or text in, image out model.

• 7:17Chris Benson?

  Right. And I think people, there's so much coming at people right now. I think, you know, we keep talking about that this year. In the five years we've been doing this podcast, we've never had a moment like the last few months where things have been coming, and new things have been coming at people so fast, new terms, new models, and people are trying to distinguish. So it's pretty, I think it's pretty fair that people are trying to make sense of how they relate together. And there's, there's a lot of connecting between, you know, the, the idea of generative and the idea of large language models overlap in a lot of areas. And you have models that are both, and you have models that are just one and stuff. But I think it's a brave new world right now in terms of the, the amount. Every show, we're just trying to figure out what matters right now, because there's a lot we're not hitting.

• 7:59Daniel Whitenack?

  Yeah. And this side of things, maybe like the 3D or video or image-based side of things, I know has its own set of kind of transformative use cases that are popping out. I even remember a little while ago, there was some technology, I think from Shopify, but others have done this as well, where maybe you have a room in your house and you want to see how you can transform it with new furniture, or something that of course you could buy. This is a real kind of e-commerce or retail sort of use case for some of this scene technology of a different kind. If you think of this sort of technology that

• 8:42Daniel Whitenack?

  can take 2D things and create these 3D scenes. Certainly, there's use cases within game development, for example, but even other cases where maybe AI has never impacted the process as much, like in real estate, for example. You know, how expensive is it to literally have a person come out with specialized camera gear? I know that we've had this in the past where it takes a special person to come out with special camera gear to capture the kind of 3D walkthrough, essentially the street view walkthrough of your house, and map that onto an actual schematic of your house.

• 9:25Daniel Whitenack?

  And here, if you imagine someone, maybe I'm now selling my house myself without a real estate agent, and I can take an app potentially and go through my house just taking 2D images and create this really cool kind of fly-around 3D view that's interactive. That's really, I think, a powerful transformative change for a number of different industries. I came across a company called Llama AI in one of the posts about this technology. I don't know exactly how much of the, if they're even using the Zip Nerf stuff, but certainly some things related to Nerf to take these 2D images.

• 10:07Daniel Whitenack?

  And they have an app that will create 3D views, which is pretty cool to see some of this kind of hit actual real users.

• 10:16Chris Benson?

  We keep talking about the fact that we've hit this inflection point where it's hitting all the, you don't have to be in the AI world, you know, for this to have a big impact. And so, you know, it's very easy, looking at the Zip Nerf video, to imagine walking around with your cellphone on an app, and you're just kind of like walking around and the app takes care of whether it's video or whether it's still images or what, and it just uploads it to this and produces this, you know, amazing, you know. So it's not your walk around that it's doing. It takes that as raw video, but then it produces this super high-quality thing. So yeah, I mean, I think this is another case where there's this one technology with thousands of use case possibilities, you know, where it just changes everything.

• 10:58Daniel Whitenack?

  Yeah, and maybe also in the, it'd be curious to know your reaction to this also with respect to kind of the industrial use cases where-

• 11:08Chris Benson?

  Oh, I've been thinking about-

• 11:09Daniel Whitenack?

  Of course, like capturing 3D scenes is very important, for example, for simulated environments where you're trying to maybe train an agent, or you even kind of an industrial training for human sort of scenario where you want to kind of take someone into an environment that it's physically hard to bring a lot of people into.

• 11:36Chris Benson?

  Yeah, or there could be safety issues and such.

• 11:39Daniel Whitenack?

  Yeah, safety issues. I don't know if that sparks things in your mind. I think in the industrial sense, this could have a more B to B sort of impact than just a consumer app.

• 11:52Chris Benson?

  Sure. I mean, a simple thing, and this is, I'm making something up in the next thing I say, but it's very easy for me to imagine intelligence agencies that are, you know, like if you go back some years to when Osama bin Laden was found, and they had, you know, various imagery and stuff, but with stuff like this, they might take all those images that they're getting from various sources and produce, you know, a high-

• 12:15Daniel Whitenack?

  Like a flyover.

• 12:16Chris Benson?

  A very, yeah, flyover and very photorealistic of certain parts of the compound where that kind of imagery, and that can be used in a military operation subsequent. Now, I'm making that up, so don't, nobody should take that as a thing. But it's not hard to imagine that. It's not hard to imagine a lot of factory uses and other industrial things where you have safety issues, you have limited access kind of concerns, where you're trying to convey that. But there's a lot of mundane things. There's a lot of home-based things and small business-based things, as you pointed out, the real estate one earlier. So- This is just one technology that we're talking about so far.

• 12:52Daniel Whitenack?

  Yeah, and I think what you're saying, it illustrates how this is impacting very large organizations, all the way down to small organizations.

• 13:03Chris Benson?

  Yeah, sole proprietorships.

• 13:05Daniel Whitenack?

  Yeah, and it's interesting how, like, if we just take this use case, for example, these kind of 3D scenes, kind of a large-scale organizations that maybe their bread and butter was either the compute associated with like rendering videos and 3D scenes, or their hardware providers that are creating specialized kind of 3D type of equipment. Like, their whole business model, they've got to be thinking, similar to other organizations that are dealing with maybe language-related problems, that are thinking about these things with respect to LLMs. There's a fundamental shift in maybe how their businesses will operate, but then, at

• 13:49Daniel Whitenack?

  the same time, it provides an opportunity for the kind of small-to-medium businesses to embrace this technology very quickly and actually make innovative products that can be widely adopted very quickly and actually be competitors within an established market. So there's an established market for 3D things that has been quite expensive over time in terms of access to that technology. So now, that whole market's going to change, and I think a lot of the players will be these kind of small-to-medium-sized businesses.

• 14:26Chris Benson?

  I agree. I think there's a moment here, that it kind of ironically, because people are so worried about like, the impact on human creativity because of all these models and stuff like that, but on a more positive note, there's this huge opportunity that you're just now alluding to, for people that if you can connect the dots as things are coming out, and you can stay on top of it, it's a great equalizer. And so, it will clearly change many, many markets that are out there, and many, many industries, and so there's huge opportunities for those who want to surge ahead at this And take advantage of that. And so I think that the message we tend to see in the media tends to be a little bit doomy and gloomy on that, but it kind of discounts the fact

• 15:08Chris Benson?

  that change isn't always a bad thing. People are afraid of it, but there's huge, huge opportunities here as well. If people choose to go find them.

• 15:33Daniel Whitenack?

  Well, Chris, there is a new llama in town. Llama two. Llama two, basically destroyed all of my feeds and concentration this week when it was released, because it is quite, to me, an encouraging thing, but also another transformative step in, in what we're doing. So llama two, for those that maybe lack the context here, Meta, or Facebook, or however you want to refer to it, Meta, had released a large

• 16:11Daniel Whitenack?

  language model called Llama, which was extremely useful. It was a, it was a model where you could host it yourself as opposed to like OpenAI. You could get the weights and host it yourself, but the original llama had a very restrictive licensing and access sort of pattern, even though you could kind of download the weights from maybe like a BitTorrent link or something like that, and those propagated. Technically, if you got those weights, you were still restricted by a license that prevented commercial use cases specifically. And now with Llama two, Meta's released the kind of follow on to Llama, and we can talk through

• 16:56Daniel Whitenack?

  some of what the differences are and what it is and some of what went into it. But I think one of the biggest things, which is, I think, going to create this huge ripple effect throughout the industry, is that they've released it with a commercial license. As long as on the day that Llama two was released, you as a commercial entity don't have greater than 700 million monthly active users.

• 17:28Daniel Whitenack?

  You can use it for commercial purposes. So maybe if my company maybe later on has 700 million monthly active users, which would be great, probably never, but

• 17:39Chris Benson?

  There'll be something past Llama 2 by then, though.

• 17:41Daniel Whitenack?

  Yes, if it does, though, I could still actually use it because it's only on the release date. So on the release date, which was was this week, if as long as you didn't have greater than 700 million monthly active users, you can use this in your business for commercial use cases. And I think that's going to have a huge ripple effect downstream. And we can talk about the model itself here in a second, but maybe just I'll pause there to get your reaction on that, Chris.

• 18:09Chris Benson?

  It made me smile when I heard that because it's kind of like saying, so long as you don't compete with us at Meta, you can use this for commercial.

• 18:16Daniel Whitenack?

  Oh, it's totally true. Yeah, like, who is that, right? So that's Snapchat.

• 18:20Chris Benson?

  Yes.

• 18:21Daniel Whitenack?

  TikTok, right? Like, you can think of, yeah, you can think of who this is. And I guess one way to put this is it's not totally open source, quote unquote. Like, we wouldn't call this maybe open source in the kind of official definition of open source.

• 18:40Chris Benson?

  Yes.

• 18:40Daniel Whitenack?

  But it's certainly commercially available to a very wide set of people.

• 18:47Chris Benson?

  Yep. You know, one of the first things I noticed when this came out on their page and they're talking, you know, there's, there's, and I'm diving into like the specifics of the model here, is we had an episode not too long ago, and you were describing about kind of the, I believe it was the 7 billion limit, you know, in terms of hardware usage and stuff. And having been taught that by you, I immediately locked in on the smallest being 7 billion there as, and I thought, ah, this is what Daniel has taught all of us about that limitation on accessibility and who can do it. So, you know, it has the 13 billion and the 70 billion size, but I definitely picked up on the 7 billion, which I'm assuming is going back to what you were teaching us a few episodes back.

• 19:30Daniel Whitenack?

  Yeah, and so just to fill in a little bit on that, so the Llama 2 release includes three sizes. So again, thinking back to what are the kind of characteristics of large language models that Kind of matter as you're considering using them. One is license. We've already talked about that a little bit here. We might revisit it here in a second. Another is size, because that influences both the hardware that you need to run it and also its kind of ease of deployment. So Llama 2 is released in 7 billion parameter, 13 billion parameter and 70 billion parameter sizes.

• 20:10Daniel Whitenack?

  And then there's also, of course, the training data and that sort of thing that's related to this and how it's fine tuned or instruction tuned. So Llama 2 is released in these three sizes, both as a base large language model and a chat fine tuned model. So there's the 7 billion, 13 and 70 billion Llama 2s. And then there's the 7, 13 and 70 billion Llama 2 chat models, which we can talk about that fine tuning here in a second. But yes, you're right, Chris, in that 7 billion, I could reasonably pull that into a Colab

• 20:51Daniel Whitenack?

  notebook and maybe with a few tricks, but with certainly with the great tooling from Hugging Face, including ways to load it in even four bit or other quantizations, I can run that, you know, on a T4, for example, in Google Colab with some of the great tooling that's out there. So not needing to have a huge cluster. The 70 billion, even with that, that's kind of another limit where using some of these tricks, I've definitely seen people running the 70 billion parameter model on an A100. Again, loading in four bit with some of the quantization stuff and all that. The 70 billion is certainly going to be more difficult to run.

• 21:36Daniel Whitenack?

  It might require multi multiple GPUs, but that's kind of that sizing range for people to have in mind and how accessible things are. And yeah.

• 21:48Chris Benson?

  How might you, I'm just curious, if you're looking at these, you're a business out there or a data scientist, and can you make up a couple of use cases that you might target with each of these where you might say, oh, I want to go 13 on this, not seven, not 70 for something like this? Can you imagine something like this? I'm putting you on the spot.

• 22:06Daniel Whitenack?

  Yeah, I think, I mean, there is certainly innumerable use cases, but I think maybe two distinctions that people could have in their mind is if you want like your own private chat GPT, right? Or like another way you could think about it is a very general purpose model. Like you could do anything with this model, like you, any specific prompt, whatever. You're probably going to look towards that higher end, the 70 billion parameter model for that kind of almost chat GPT like performance. You're going to have to go much higher. But as we've talked about on the show before, most businesses don't need a general purpose model. They need a model to do a thing.

• 22:49Daniel Whitenack?

  And so, or a task or a set of tasks. And so in that case, I think businesses, because this is open and commercially licensed, businesses that could take those seven and 13 billion parameter models and fine tune them for a task in their business, which also is increasingly has amazing tooling around it. Again, from, from hugging face and others with the PEF library, parameter efficient fine tuning. And the LoRa technique, which is the low rank adapter technique, which basically only adapts an existing model. It's kind of an adapter technique rather than retraining a bunch of the, the original model.

• 23:33Daniel Whitenack?

  This opens up fine tuning possibilities in these smaller models where that fine tune for an organization is going to perform probably better than any general purpose model out there. And because it's that smaller size, you can run it on a reasonable set of hardware that's not going to require you to, you know, buy your own GPU cluster to host the thing, right? So that's kind of a, maybe a range of use cases that people could have in mind.

• 23:59Chris Benson?

  I have one more question for you before we abandon this. Seven billion to 70 billion being an order of magnitude jump on that. Why would you have something fairly close to that at 13 billion parameters? Like what's the difference in seven and 13 when the next step is all the way up to 70? Well, what, what's the rationale you think?

• 24:19Daniel Whitenack?

  Yeah, so it is interesting, actually, if I'm understanding right from some of the sources that I've

• 24:26Daniel Whitenack?

  That I've been reading, there was actually a, I forget if it was 30 or 34 billion parameter model that they were also had in pre-release and were tuning. So there was another one that kind of fit in that slot that is kind of missing that gap like you're talking about. Yeah. Like if you think of MPT, MPT has a 30 billion parameter model that fits in that kind of gap. My understanding, and you know, if our listeners can correct me if I'm wrong, please do, but my understanding is that they actually did test that size of model and found it to not pass their kind of safety parameters around harmful, potentially harmful

• 25:07Daniel Whitenack?

  output or not truthful output, that sort of thing. So they decided actually to hold that back. So it could be possible as they instruction tune and get human feedback, potentially more iterations of reinforcement learning from human feedback. There may be a model that they release in that parameter range. So that was one thing that, that happened, I think. It is interesting, you know, several different things here that are unique about this model specifically, or maybe the release as well, other than the license, is they were fairly vague on the data that went into the pre-training. So they talked specifically

• 25:52Daniel Whitenack?

  about some very intense data cleaning and filtering that they did on public data sets. And it was trained on more data than the original Llama, but they're fairly vague on the mix of that data and all of that. So that may be related to feedback they got on the data sets that were used in the first Llama. I don't know, but the technical paper was mostly related to the, the modeling and fine tuning trickery and methodologies that they used, which was interesting. And one of those interesting

• 26:29Daniel Whitenack?

  Elements of the way that they fine tune this model was, I think, the reward modeling. So if you remember, like, the GPT family of models, the MPT Falcon, these different models, one of the things that is often done with these models is this process of reinforcement learning through human feedback, which is this process, and we covered this on a previous episode, which we can link in the show notes, but actually using human preferences to score the output of a model and then actually use reinforcement learning to correct the model to better align with human preferences or human feedback.

• 27:09Daniel Whitenack?

  They actually used two separate reward models in this fine tuning of the chat-based model, one that was related to helpfulness, and then the other one, which was related to safety. And one of the interesting things that they talked about in the paper was how sometimes those things can kind of work against each other if you're trying to do both of them at the same time. So they actually separated out the reward models that they used for the chat fine tuning into these two reward models, one for helpfulness and one for safety, which is quite interesting, I think.

• 28:01Daniel Whitenack?

  So Chris, maybe just a couple other things related to Llama, and then I want to see your feedback on Code Interpreter as well, because we haven't talked about that yet on the show, and maybe Claude 2 if we can get to it.

• 28:15Chris Benson?

  Yeah, we got to mention Claude 2 as well, because they were both big releases.

• 28:18Daniel Whitenack?

  Yeah, so just one maybe other note, which I find quite interesting, and actually, I'd love our previous guest Damian's thoughts on this, who was in our last episode about the legal implications of generative AI. But one of the interesting things about the Llama license, in addition to it allowing this commercial usage... Is that there is a technically a restriction in the Llama license that says you will not use Llama materials, which includes the model weights and etc. Or any output or results of the Llama materials to improve any other large language model excluding Llama 2 or derivative works thereof.

• 29:02Daniel Whitenack?

  So essentially what this means is if you're using Llama 2 and you want to fine tune a model or you're fine tuning a model off of Llama 2 outputs, you're stuck with Llama 2. Basically, Llama 2 is your model and that you're going to stick with Llama 2. So you couldn't, for example, technically take the Llama outputs from Llama 2 and fine tune, say, Dolly 3 billion, right? That would not be allowed by the license. And of course, that's something that people are doing all over the place. They're taking outputs from GPT-4 and fine tuning a different model or taking outputs

• 29:43Daniel Whitenack?

  from a large model like, you know, maybe Llama 2 70 billion now and fine tuning another model that's smaller based on a certain type of prompt or something. So this is restricting that family of models that you're allowed to do that sort of thing with, which I is the first time I've seen that and I think it's kind of interesting.

• 30:06Chris Benson?

  Yes, it strikes me as a as another Mark Zuckerberg anti-competitiveness, you know, thing, which which he's fairly famous for. I mean, that's kind of even before this.

• 30:15Daniel Whitenack?

  Yeah. And how could you enforce such a thing?

• 30:18Chris Benson?

  Yeah. I that was my next question to you is, is there any possible way that you could conceive of to actually know that from an enforceability standpoint?

• 30:28Daniel Whitenack?

  I have no idea.

• 30:29Chris Benson?

  I don't either. So it seems it's like it's a license thing and it will concern the lawyers, but it's hard to imagine. I mean, going back to our conversation last week, once you have output and that output is input to more output and, you know, there's a point where it becomes very, very, very difficult to know what the what the sourcing really was.

• 30:50Daniel Whitenack?

  And the fine tunes are already appearing off of Llama 2. So the most notable probably is Free Willy, which is from Stability AI and is a fine tune of the largest 70 billion model. But there's other ones coming out as well. And so I think we're about to see just a huge explosion of these Llama 2 based models for a whole variety of purposes. And who knows how they will fit into that licensing restriction, or how open people will be about that. But it's about to start. The fine tunes are already coming.

• 31:27Chris Benson?

  Yeah, well, you know, to your point earlier, they weren't terribly clear about the data that they were sourcing from their own standpoint.

• 31:34Daniel Whitenack?

  Yeah.

• 31:35Chris Benson?

  And I find it interesting, a little ironic.

• 31:38Daniel Whitenack?

  A bit of a double standard, maybe?

• 31:40Chris Benson?

  Yeah, a little bit of a double standard right there in terms of like, we're not going to tell you everything about how we're doing input. But by the way, you better not use our output for your, you know, for others.

• 31:47Daniel Whitenack?

  Yeah, yeah.

• 31:48Chris Benson?

  So yeah, a little interesting. Do you think there's any risk of a walled garden kind of concept happening in large language models if others were to follow this lead on anti competitiveness?

• 32:01Daniel Whitenack?

  Yeah, it will be interesting. I think it is a notable trend that the first Llama from Meta was not open for commercial at all. And now they're opening it up for commercial purposes. And, you know, maybe there's a separate trend that will happen with some of these use based restrictions that people are importing into their licenses and how useful those things are over time, that will may shift and we'll see those things die off. Or maybe if they're enforced and there's precedent, maybe we'll see something go the other way. I'm not sure. But speaking of models that you might get their output and use it to train other models, that is these large scale proprietary closed models from

• 32:45Daniel Whitenack?

  people like OpenAI and Anthropic and others. We've got a couple of things that we haven't talked about on the show yet, which people should probably have on their radio radar. One of those is Claude 2. What do you think about Claude 2 from Anthropic?

• 33:03Chris Benson?

  Yeah, I've been I've been playing around with it a lot in the last week. And I kind of have a set of things that I try over and over again. They're kind of my standard tasks as new models come out. And some of them are coding and some of them are content generation, which are kind of the two big things that I use most often. It was interesting, you can put, you know, the input size for Claude two is much larger than the others.

• 33:25Daniel Whitenack?

  Like much, much larger.

• 33:27Chris Benson?

  Much, much, much larger.

• 33:28Daniel Whitenack?

  So 100,000 tokens.

• 33:30Chris Benson?

  Yeah. And so it's had me kind of change the way I'm approaching it in that by contrast with like chat GPT. And you're trying to figure out with with the limits that you have both on input and output. How do you kind of prompt engineer your way to get, you know, where you're trying to go? Which has become this whole skill set we've been talking about, you know, in recent months. And yet Claude two almost kind of wipes that out a little bit in some ways, not not in always. And that you can hit it with a much larger input space. And and so it's changing how I'm thinking about kind of getting to the output that I want. And the output is a bit different. It's not the same. I'm getting out different outputs from from all the models. So

• 34:11Daniel Whitenack?

  Yeah.

• 34:11Chris Benson?

  They're not all the same, definitely. I think my biggest thing is with all these new releases, I'm trying to figure out how do I use each one? When do I I'm trying to develop my own strategy on when do I go to chat GPT by default? Like, when's that the right thing? And that's changing as we'll talk about with things like plugins and stuff that's evolving. But then Claude two comes out and then you have, you know, on the open source side, as we just talked about with Lama two. So I think trying to understand all the tools in the toolbox in relation to each other has been interesting. So Claude two, I'm I'm really focused right now primarily on on large content output is kind of where I've landed on that.

• 34:47Daniel Whitenack?

  And the 100K context length of Claude two is something I find really compelling as well. There was also a significant paper that came out that caused a lot of waves in terms of context length and thinking about that. Which showed kind of as you increase context length, you lose any significance of the middle bit of that context. So the beginning and end is more important in terms of what makes the output of the of the model quality or not in terms of how you would measure that. And so we'll link to that paper maybe in the show notes as well. But I've tried some things. I mean, I don't know exactly all of the details again.

• 35:32Daniel Whitenack?

  Claude is one of these closed models, so I don't know all of the details of how they're doing things. And because it's sitting behind an API, it's hard to know how those things evolve over time. But for example, I took one of the things with Claude 2 is I just took one of our complete podcast transcripts. So a full episode, so 45 minutes of audio transcript. I took episode 225, which I really enjoyed talking a lot about the things that I'm working on right now with Prediction Guard and just asked it to give me a summary of the main takeaways and, you know, paste it in the whole thing and it's like a fairly good, comprehensive takeaways. Like many companies ban usage of certain LLMs, blah, blah, blah.

• 36:16Daniel Whitenack?

  You know, Prediction Guard is trying to provide easy access, structuring, validation, compliance features for LLMs, making LLM usage easier, blah, blah, blah. And it gives these great takeaways. And then I asked, you know, hey, suggest a few future episodes that we could do that maybe cover related topics, but things that weren't covered in this episode. Pretty good. Some of them are kind of generic, right? I'll look at current state of AI agents and automation. How close are we to no code AI app generation, blah, blah, blah. So that all kind of off of this large context of the transcript input was quite interesting.

• 36:58Chris Benson?

  I'm curious. I'm going to put you on the spot also. As someone who's working on your own product, and I know this is not a Prediction Guard episode, but I'm asking on my own behalf and on behalf of the listener, how do you, as someone who is looking at these different models, how do you think of those different models? How do you kind of structure them in your mind in terms of what you're offering? You've been evolving rapidly over the last few months and I'm always curious to see kind of where your head's at on this now as you're looking at them.

• 37:23Daniel Whitenack?

  Yeah, I think the things consistently that I'm seeing are that I made a post on LinkedIn about this as well. Even my own applications that I'm building, LLM-based applications, having access to multiple models rather than a single model, I think is a really nice usage pattern where if you, the easier we can make it, and there's, there's other people that are doing this as well, other, you know, in Prediction Guard, you can query a whole bunch of models at the same time concurrently. There's other systems that will let you look at that output as well. So nat.dev and some of the toolbar stuff

• 38:06Daniel Whitenack?

  that Swix is doing. We had a collaboration with him and the Laten Space podcast. So the more you can tie these things together and look at the output or automatically analyze the output of multiple models at the same time, I think that's really useful because it, it's hard to generally evaluate these models until you start evaluating them for your use case and building intuition about them for your own use case. So I think the pitfall that people maybe fall into is saying, oh, I'm gonna use this model before they've even tested that for their use case, right? Try creating a set of evaluation examples for your own use case and then try out a bunch of different models for that. And also

• 38:48Daniel Whitenack?

  try out the things that are becoming more standard kind of operating procedures for building LLM applications, like looking at the consistency of outputs, running a post-generation validity or factuality check on the output. So checking a language model with a language model, doing input filtering and all these sorts of more engineering-related things. So those are some of the things that I'm seeing, but having access to a bunch of models at the same time, I think is a Is something that can really boost your productivity.

• 39:25Chris Benson?

  I appreciate that. And to our listeners, we're not making it a prediction guard show or episode, but as a co-host, Daniel's excursion through this and his professional has made him, in my view, one of the world's true experts in how to look at all these together. And since we have the benefit of him co-hosting the podcast, I'm going to continue to take advantage of that expertise for all of us.

• 39:48Daniel Whitenack?

  Thanks, Chris.

• 39:49Chris Benson?

  Sorry about that, Daniel. Sorry for putting you on the spot.

• 39:52Daniel Whitenack?

  Yeah, no, no worries. I think the other thing maybe to highlight with Claude 2, and something that you were talking about in chat before we jumped into this episode, was Claude 2 versus, or maybe Anthropic and their offerings versus OpenAI. How do we understand that? Like, how do we categorize these things? I think one of the interesting things with Claude 2, so we've seen both Anthropic and their Claude models and OpenAI and their GPT models increase context size over time. GPT models, not quite as far as Claude, but both have increased. They've also both added in some of this functionality, which I think is very interesting.

• 40:35Daniel Whitenack?

  Claude 2, I think first, if I'm not wrong, the ability to add in your own data. So in Claude 2, there's a little attachment button, and you can upload PDFs or text files or CSVs and have that inserted into the context of your prompt, which I think is, of course, extremely powerful. We've talked about adding in external data into generative models and grounding models in the past. It's very powerful. Now, OpenAI is doing this in a slightly different way, and I think this is something worth calling out on the podcast is, with their new code interpreter beta feature within ChatGPT,

• 41:15Daniel Whitenack?

  you can upload data, but it's processed through the code interpreter in a different way than what Claude is doing. So we all know that ChatGPT and GPT models can generate really good code, and specifically, good Python code. And so what OpenAI has done For their kind of data processing agent within ChatGPT has said, well, let's just have our model generate Python code. Then we'll hook up the ChatGPT interface to a Python interpreter and just go ahead and execute that code for you over your data and then give you the the output. So this is maybe a distinction that people can have in their mind.

• 41:58Daniel Whitenack?

  Claude 2, you can upload this huge amount of context, you can upload files, insert it into the prompt. As far as I know, they're not running any kind of code interpreter type thing under the hood. ChatGPT might not be inserting all of that into the prompt, but they're actually saying, well, what if we decompose what you're wanting me to do with this external data into something that can be executed by a sort of agent type of workflow where you upload your data and ask me to like do some analysis over it. I'm going to generate some code. So the language model generates some code and then that code is actually executed in the background.

• 42:41Daniel Whitenack?

  It returns a result, which is then fed back through a model to give you generated output back in the interface. So it's actually a multi-stage thing happening in code interpreter in OpenAI.

• 42:53Chris Benson?

  It effectively produces a no-code solution, you know, where you get an output and you're just kind of skipping the whole thing, you know, instead of instead of using the language model to generate your own code and to be your code assist and all that, and then you're still doing it. It's kind of skipping that whole step right there.

• 43:09Daniel Whitenack?

  Yeah, and I can give an example I actually ran prior to the show. So I have Claude and the OpenAI code interpreter side by side open. I uploaded a file with a bunch of Yoruba, which is a language in Africa, transcriptions out of audio, which are from the Bible TTS project that we worked with Koki and Masakane on. And so I uploaded this file, which includes this Yoruba text in a CSV format. OpenAI said, great, you've uploaded this file, let's start. By loading and examining the context and then it has this sort of show work button.

• 43:48Daniel Whitenack?

  And you can see the actual code that it generated, which is Panda's code to import the CSV and then output some examples. And so you can expand that and actually see the code that it ran under the hood and the conclusions that the agent came to. And then I asked it, okay, well, plot the distribution of the transcript links. Are there any anomalies? And then again, it says, hey, show work. And you can see it's importing map plot lib. It's taking in the CSV. It's actually creating the plot and actually generates an image out of the transcripts. It says, I didn't find any anomalies. They're all kind of within the same distribution. There's not any anomalies. Then I asked it, can you translate all the Yoruba to English?

• 44:33Daniel Whitenack?

  And that's where it ended up stopping because it said, no, I'm not good at doing that. And Claude actually stopped there as well and said, no, I'm not going to do that. I also uploaded the Yoruba alignments to Claude and it said, hey, sure, let me analyze these transcripts. And it just output some general, like there are 50 audio links, the transcript links. There's no Python code there. It just gave me some takeaways, right? And then I said, are there any anomalies? And it said, I checked and I can't find any. And could you translate it? And it said, unfortunately, I can't. So it's all still a chat based thing. So you can see kind of different approaches to this complicated workflow

• 45:15Daniel Whitenack?

  of having almost an assistant agent executing code for you versus putting more context in the language model and having it reason over that context.

• 45:25Chris Benson?

  So they're almost getting their own strengths at different types of approaches to problems. Would that be fair?

• 45:32Daniel Whitenack?

  Yeah.

• 45:32Chris Benson?

  So that's another way of thinking about it is, as you start understanding how the different large language models approach a problem and the tooling that might be better or worse for a given use case. That also will help you kind of pick which way you want to go in addition to maybe just using multiple models as you talked about earlier.

• 45:50Daniel Whitenack?

  Yeah, exactly. And there's so much to dive into on all these topics that we've covered today. I'm going to make sure that we include some really good learning resources for people in the show notes. So make sure and click on some of those. There's a guide from DataGen on the neural radiance field stuff, the Nerf stuff, that you can learn a bit more about that. There's a hugging face post and a Will Schmidt post on Lama 2 that are both really practical, kind of how do you run it? How do you fine tune it? What does it mean? And then there's a nice post from the one useful thing, Ethan Mollick blog or newsletter about code

• 46:32Daniel Whitenack?

  interpreter and how to get it set up and some things to try. So we'll link that in our show notes and I think people should dig in, get hands on with this stuff. Things are, things are updating quickly and the only way to really get that intuition about things is to dive in and get hands on.

• 46:47Chris Benson?

  It is. It's the most interesting moment we've had in the AI revolution of recent years. And just so much cool stuff right now. Anyway, thank you for taking us through all the, the understanding and explanation of these things.

• 47:01Daniel Whitenack?

  Yeah, definitely. It's, it's good, a good time. Hopefully, people enjoy the rest of their week and maybe go see Oppenheimer or Barbie, depending on your, which of those is most interesting to you. But we'll see you next time, Chris.

• 47:15Chris Benson?

  See you later. Thanks.

• 47:20Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.