Mamba & Jamba

First there was Mamba… now there is Jamba from AI21. This is a model that combines the best non-transformer goodness of Mamba with good ‘ol attention layers. This results in a highly performant and efficient model that AI21 has open sourced! We hear all about it (along with a variety of other LLM things) from AI21’s co-founder Yoav. Sponsors: Fly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Changelog News https://changelog.com/news – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today https://changelog.com/news. Featuring: Yoav Shoham – LinkedIn https://www.linkedin.com/in/yoavshoham, X https://x.com/yshohamChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Jamba - A Groundbreaking SSM - Transformer Open Model https://www.ai21.com/jambaAI21 Labs https://www.ai21.com/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of the Practical AI Podcast. My name is Daniel Whitenack. I am CEO and founder at Prediction Guard, and I'm joined, as always, by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How are you doing, Chris? Doing great today, Daniel. How's it going? It's going great. The sun is out and summer is upon us, along with lots of new AI models and excitement going on in the space. And on that note, specifically as related to large language models, we're really excited to have with us today Yoav, who is the co-founder

• 1:23Daniel Whitenack?

  and co-CEO of AI21 and Professor Emeritus at Stanford. Welcome, Yoav. How are you doing?

• 1:30Speaker 02

  I'm doing good. Really a pleasure to be with you guys.

• 1:33Daniel Whitenack?

  Yeah, yeah, we're so excited to have you on. It's a show we've been wanting to have for for some time now. I'm wondering if you could kind of give us a little bit of the background of AI21 and specifically, maybe how you view AI21 as fitting into this wider landscape of LLM companies and technology?

• 1:58Speaker 02

  So maybe a good starting point will be to say why we started the company in the first place a little over six years ago. We started the company because we believe that deep learning, remember at the time L L Ms were not a thing. But deep learning was mostly applied to vision. We believe that modern day A I requires deep learning. It's a necessary component, but not sufficient. We believe that certain aspects of intelligence, this thing we often call reasoning, will not emerge purely from the statistics. And it's the sort of thing A I did back in the in the eighties. And we believe that we left money on the table and it's time to bring the two together. That's why we started the company.

• 2:41Speaker 02

  Uh, now fast forward today, where, what does the landscape look like and where do we fit in? So although I said that large language models, so very quickly we fell into L L Ms. Uh, we, uh, we're the heaviest users of G P T three when it came out. Uh, we decided to roll our own and really language is where the action is because we often say that machine vision is a lens into the human eye, but language is a lens into the human mind. Because there's no thought as intricate and nuanced as you want, that can't in some way be expressed in language. Uh, vision is a quote unquote easy problem. Of course it's not easy, but there's something to understand that this is a phone.

• 3:24Speaker 02

  I don't really care what the pixel is way on the on on side here. Always exactly true, but it's really primarily true. That's not true with language. Language connections matter terribly. You change a word here, the whole meaning of the sentence changes in general. You can't escape semantics when you deal with, uh, with language. And so it's harder, but if you crack it, that's gold. Uh, if you look at the enterprise and from the beginning, we were focused on the enterprise. Eighty percent of the data in the enterprise is text, mostly, um, either not used or way underused. And there's a really good opportunity there and that's kind of been our focus. So of course, we're not the only people with large language models.

• 4:08Speaker 02

  We are one of the handful of companies that do really large, very capable language models. Our first model was called Jurassic One, going back a few years. It was not the most innovative model, but it was a good workhorse. It was a GPT-like autoregressive left-to-right model. And at the time, it was slightly bigger, slightly better than GPT-3. Of course, both those models have by now eclipsed. We very recently released our most recent model called Jamba, which is very interesting in a number of ways. And we can dig deeper, but maybe at, you know, thirty thousand feet. Architecturally, it's different. It's not pure transformer model.

• 4:53Speaker 02

  It really is mostly based on structured space, structured space state model, SSSM, as they're called. And we can speak about the advantages and disadvantages of those. But basically, we took that architecture and added elements of transformers, the attention layer, to get the both-to-boss worlds. And you get performance that is as good as any model of its size, better than most of its kind of size group, and extremely efficient. We have a context length that's larger than any other model of the size. Like, you know, we released, the version we released has a 250K context window length, although we trained it up to a million.

• 5:35Speaker 02

  And yet, it all fits onto a single eighty gigabyte GPU. And so your show is titled Practical AI. This starts to make it practical.

• 5:46Daniel Whitenack?

  That's great. And speaking of practicalities, you mentioned the focus on enterprise from the beginning. You also mentioned that a lot of data in the enterprise is kind of locked up in this unstructured text. I remember when I first got into data science, the focus is, oh, we're gonna do big data and all of this cool analytics stuff with data warehouses. And I think that's sort of waned a little bit. I'm wondering if you could talk to that point. Like, why are enterprises? What types of value can they get out of this sort of text that's sitting around? And because I think maybe a lot of listeners, maybe they've tried, you know, these chat interfaces, whether it be ChatGPT or

• 6:31Daniel Whitenack?

  Gemini or whatever, but maybe they're less exposed to the workloads that enterprises are doing with L L Ms. So could you give us a picture of how enterprises are unlocking value with that kind of 80% of text data, maybe just by way of example or at a high level?

• 6:50Speaker 02

  Sure. And really, the use cases are quite broad. The industries are very broad, whether it's finance or healthcare, education, or you know, you name it. And the use cases vary are varied. But to pick some concrete ones, let's say you have manuals, there are companies with thousands of manuals. And whether it's the end user wanting to, I recently needed, I had a new sort of oven microwave combination. And for the life of me, I couldn't find the relevant information in the manual. So I searched online and so on. It'd be really convenient to go and answer a question and get just the right answer. But even if it's not the end user, it could be the tech support person who themselves want to get quick answers.

• 7:35Speaker 02

  So that's the example. We call this contextual answers. Another would be summarization rather than response to a specific query. You have this 10 K report that came out and you want a pithy summarization of it, maybe a summarization geared to a certain aspect you care about. So that'd be another use case. These are both a wave of consuming data. There's of course, Gen AI is a terrible name, but we won't find that battle.

• 8:03Daniel Whitenack?

  You're stuck with it.

• 8:05Speaker 02

  Well, you know, you'll get me started. I'll start complaining about Gen AI, about AGI and so on. But certainly some use cases call for producing information, not only consuming information. So for example, one of our use cases, very successful, our product descriptions, you have companies, retailers and you know, e-commerce companies who have thousands of products that come online constantly. And writing a product description is labor intensive, error prone, expensive, time consuming.

• 8:35Speaker 02

  And we're able to compress all of that dramatically. So these are some use cases.

• 8:42Chris Benson?

  I'm kind of curious also, as you're looking at these opportunities in the enterprise and addressing these various use cases, as a company who is creating models and putting them out there for enterprises to use, for people who have not, you know, are not in the industry itself, how do you, as a co-founder and CEO, see your company as like, how do you say, let's go do this? Like, we see the value in this compared to others that are making models? Like, in other words, if you say, I'm going to make a model, what is it about that motivation which makes you think you'll make a difference in that enterprise market? You know, and you're kind of representing all companies that do so, just to shed some insight on on how a founder thinks in the space?

• 9:27Speaker 02

  I wouldn't purport to represent the entire industry, so I'll speak for ourselves.

• 9:32Chris Benson?

  Fair enough. Overshot on my asking. No worries.

• 9:36Speaker 02

  But maybe some of these comment others. So first of all, the baseline is a general purpose, very capable model. There's a need for that. Now, there are companies that provide services using other people's models, and that's totally legit. If you actually own the model, that you can do things that you wouldn't be able to do otherwise. And our emphasis, in addition to the general capability of the model, is in order to make it practical, there are two things, especially in the enterprise. So if you're, you know, using a chatbot to, you know, write a homework assignment, the stakes are low. A mistake doesn't carry a

• 10:16Speaker 02

  big penalty, and probably won't somebody, nobody would read it anyway. But if you're writing a memo to your boss or to your prize client, and if you're brilliant 95% of the time, and garbage 5% of the time, you're dead in the water. And so reliability is key, and as we know, White language models are these amazing, creative, knowledgeable system, but probabilistic. And so you will get, I don't like, here's another term I don't like hallucination, but you'll get stuff that isn't grounded in fact, doesn't make logical sense, and so on. And so you can't do that. You need to get high reliability. That's number one. I'll tell you in a moment how we do that. But the other thing, it needs to be efficient.

• 11:02Speaker 02

  You know, for every customer query, you're going to pay $10 to answer it. It'll take you 20 seconds to answer it. That's not no good either. And so you need to address that also. So we have several things we're doing in this regard. The first is what we call task-specific models. In addition to our general-purpose model like Java that came out, we provide language model that tailored to specific use cases. You can think about it as a matrix. You have industries and you have use cases. And it turns out that while initially some, you know, you might think that, oh, I'm going to do a healthcare LLM or a finance, that's a little bit boiling the ocean. You want to be more specific. And one way to be specific is to think about what I'm going to use it for.

• 11:47Speaker 02

  These are the columns. So for example, take summarization. That's a specific task. And now you can optimize your system. And I am deliberately saying system and not language model. I'll tell you in a moment why. But you can optimize that for that use case. So all companies now are experimenting with multiple solutions, as they should. And in this particular use case, a very large finance institution took several of their financial documents, several hundred, and tested various solutions. Our task-specific model on summarization and some of the general-purpose models of, you know, other companies. And ours were just hands down better in terms of the quality of the answers they got.

• 12:31Speaker 02

  There was no hallucination, if you pardon the expression, very on point, very grounded and so on, because it optimized for the task. But by the way, If the system is a fraction of the size of the general purpose model. So you get the answers immediately and the cost of serving is low. And this enables use cases that this latency and unit economics enable use cases that would just be unrealistic otherwise. So our task specific models are one approach. And maybe I won't overload my answer with saying why it's not only models, but we'll get to AI systems. The other is, and it's related, having models are highly efficient.

• 13:16Speaker 02

  That goes to Jamba as an example of a model that's very capable, but not big. If I were to jump ahead and, you know, let's think about 2024. What are we going to see in this space? Among other things, you'll see focus on total cost of ownership of the reality of serving these models. You're going to see a focus on reliability. And you're also going to see focus on, that's a term I hate, agents. But AI systems that are more elaborate than this transactional interaction with a long term of tokens in, you know, a few seconds token back. Thank you, on to the next one. More elaborate. So this is the thing what's going to happen technologically in the industry.

• 13:59Speaker 02

  You're also going to see correlated to that, the industry move from what today's mass experimentation to actual deployments. We're seeing signs of it now and I think 24, you'll see this sort of face shift there also.

• 14:31Speaker 00

  This is a change log news break. On April 18th, Meta released the latest version of their open-ish large language model with state-of-the-art performance. The Verge rounds it up like this. Quote, "Meta claims both sizes of Llama 3 beat similarly-sized models like Google's Gemma and Gemini, Mistral 7B, and Anthropic's Claude 3 in certain benchmarking tests. In the MMLU benchmark, which typically measures general knowledge, Llama 3-8B performed significantly better than both Gemma 7B and Mistral 7B, while Llama 3-70B slightly edged Gemini Pro 1.5."

• 15:13Speaker 00

  End quote. What followed was your typical X-bros posting "N mind-blowing demos of what Llama 3 can accomplish," where "N" equals the number that a rival X-bro just posted, plus one. Not very interesting, but two things that did stand out as interesting to me about this announcement. First, they didn't compare Llama 3 to GPT-4 at all, so we can only assume it still comes up short when compared to OpenAI's best. Second, they continue to call Llama "open source" even though the license retains the commercial requirement of your business not being too big, which is 700 million monthly active users. So I guess Llama 3 is open for businesses of all sizes, depending on how you define "all" and "sizes."

• 15:58Speaker 00

  You just heard one of our five top stories from Monday's ChangeLog News. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com/news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com/news.

• 16:24Daniel Whitenack?

  So Yoav, I love that you bring in this element of thinking about AI systems, not just large language models or the model. Maybe that ties a little bit into what you were just talking about, about more complicated workloads or automations that are likely coming as part of the solutions that people are building. But I'm wondering if you could comment on that, like where, where does systematic thinking and the thinking about architecting AI systems fit within what you're seeing people do now and what you think needs to happen for them to get? Value out of these models?

• 17:01Speaker 02

  So the part of the answer that I'm comfortable speaking about are has to do with what is out there already. And the others, I'll speculate maybe at a little more higher level. So even if you look at a task specific model, they're really not models. They're little systems. So when you say one of your summarization and you say, I care about these elements. There's a little data processing and reasoning goes on before you call the language model. So you feed it, you don't just stick it into the context. You actually do some reasoning so you can steer the model the right direction. And then when you get something back, you don't just spit it out. You don't sort of sample temperature zero and give the top answer. You get answers and you evaluate them with validators.

• 17:46Speaker 02

  And only when you're confident that the answer is legit, you return to the user. And it may sound very expensive, but actually the operation of an L L M totally dominates in terms of the compute resources and time, these other elements. And that's an example of a system around the language model. But that's a baby step. What you're going to see is, and you're already seeing it now, but right now it's people touching parts of the elephant and doing it in a very ad hoc way. But you're going to see people stitching together multiple calls to a language model because the task may require multiple things. And it's not just chaining. It can be more complicated scripts that you're running.

• 18:31Speaker 02

  But you can't just do it. It's not like writing a, you know, a script, you know, a script, a familiar scripting language and running it because the computing elements here are different. They're expensive and they're error prone. And if you just, for example, cascade calls to a language model, number one, it can be very expensive. And second, these errors compounds and you get at the end much more noise than signal. And so you need to worry about that. You need to execute differently. And so that's an example of. What you'll see, and there are other aspects of these AI systems that you'll see come into play. The term orchestration is often used here.

• 19:14Speaker 02

  It means different things to different people, but very much you have these elements that are running either sequentially or in parallel somehow, you need to execute this execution, kind of like an operating system, but an operating system with AI elements. And so we and other people use the term AIOs. Again, an overloaded term doesn't mean anything precise, but that's the spirit of things.

• 19:38Daniel Whitenack?

  I kind of want to get maybe to the roles that are interacting with this AI OS, because I think one of the things people are struggling with is, how do I put the right talent in place to build these? Because you're talking about like programmatic, operational, systematic thinking, which is kind of like there's an element of engineering there, but it's not people that are necessarily building their own models. They're architecting these solutions and putting the right checks, the right validations in place. They're creating, you know, more than chains, these workflows. And there's some engineers coming to the table there,

• 20:18Daniel Whitenack?

  but there's also domain experts who maybe are able to speak into some of how the models are prompted. So do you have any kind of observations from your experience with how people are putting together teams to architect these solutions? And these systems like you've just described, is it, from your perspective, still going to be a heavy kind of engineering dominated type of process going forward? Or are you seeing a mix? What's your observation there?

• 20:50Speaker 02

  So my answer won't be based on an observation because the systems don't exist yet. They're baby solutions right now, but I don't think they represent what we'll see going forward.

• 21:03Speaker 02

  But in answer to your question, it very much will be a mix. There will be companies such as ours that will put in the foundational infrastructure to run these complicated flows. These will have to be extensible systems and they'll be extensible in a variety of ways. Some of them, absolutely, you'll be able to have programmers write actual code and insert the code there, but there absolutely will be a role for low code or even no code specification of the flow you want on top of this framework. There will be a data scientist that will write validations of various kinds and data pipelines for sure.

• 21:48Speaker 02

  And so I think everybody from the developer to the data scientist, to the business user who's somewhat savvy, to the end user who just wants a system that works, everybody will have a role and interaction. And we haven't mentioned DevOps yet. DevOps here is going to be very important also.

• 22:08Chris Benson?

  As we've kind of talked around the ecosystem a little bit and what, you know, about systems themselves, can we turn a little bit? And could you tell us a little bit about, as we're leading toward Intijamba, but I'd like to know a little bit about kind of where the company has been and some of the models that you have have put out there leading into this one and kind of the heritage of how you've developed that. We'd really be interested in kind of how you've pursued that since you started the company.

• 22:35Speaker 02

  I can divide it into three periods in our long history of six years.

• 22:42Chris Benson?

  That's an eon in AI these days, you know, that's.

• 22:45Speaker 02

  I had a different color hair when we started. As I said, we started by building Jurassic One. We just felt like we absolutely had to build it and we did, we innovated there, but in a minor way, we, you know, we had a vocabulary that was five times the size of what was common at the time. It was rather than 50,000 tokens, we had 250,000, slightly larger than GPT-3, not to make a point just because it worked out that way, 178 billion parameters, a dense model, and that served us well. But the next phase in our sort of, we did many things. We had our own application called Wordtune that had done very well, a reading and writing assistant using our technology.

• 23:29Speaker 02

  But on the models themselves, the next thing we put out are task-specific models, which basically is not really distillation and it's not just fine-tuning. Like I said, it's putting a system around it, but at the end of the day, you get something compact for certain use cases and that set is growing. That was our second phase. And the third phase was really seeking a way to make these models fundamentally more scalable, more efficient to serve, especially in this era of, you know, RAG kind of solutions. So you have stuff that you want to kind of bring in at inference time to influence the output of the system.

• 24:12Speaker 02

  And at some point, the system chokes, you know. We had a context window of 4K, then 8K, then 16K. Now, although some big numbers are thrown out, but most models choke at 32K, maybe 64K. That's not enough if you want to put it. So we wanted something that, now, if you were to run it on, you know, 64 H100s, you can do a lot of things, but that's not realistic. So the question was how to get something that's efficient, that can run effectively on a small footprint, and that's how we got to Jamba.

• 24:46Daniel Whitenack?

  With Jamba, you mentioned taking some things from kind of the Mamba architecture, this sort of SSM and adding in some transformer-based things. For those that aren't familiar with the kind of background with those types of models, maybe the kind of non-transformer models that people were exploring. Could you give a little bit of context to that and why it was important for, I mean, you've already mentioned efficiency and other things, but why you felt it was kind of important in this generation of model to pull the trigger in a slightly different architectural direction?

• 25:24Speaker 02

  Sure. And for this, maybe we could double kick a little bit about how these systems are architected. So at some point, the dominant architecture where the RNN, the, you know, these, and then LSTMs, as you go left to right, the system doesn't remember the distant past. What it does, it carries with it the state that somehow encapsulate everything that it's seen so far. That's quite powerful, but as this pass gets long, it gets harder and harder to encode and access that information that's been coded. And it worked fine for vision, because this in vision, object recognition is something very local.

• 26:06Speaker 02

  It's iconic, iconic in the sense that what you see is what you get, right? Like I said, the phone, this is a phone, I don't care what's here. So I go along, I hit the phone, so I don't need to remember, but in language, different. And in fact, if you looked at the benchmarks, by the way, another pet peeve of mine, benchmarks are, can be very misleading, but that aside, if you looked at the National Image Benchmark, they kind of putted along with not much progress until transformers came in. And transformers, again, coincidentally, what is it, about six years now? They changed the architecture and they had the attention mechanism that says, no, I mean, as I'm going along, I can relate disparate pieces of information. And that allowed you to do things you couldn't do otherwise. And that's great.

• 26:51Speaker 02

  So the quality answers, shut up. You pay a price, because the complexity is quadratic now in the context length. And that kills you, which wasn't the case with RNNs or LSTMs. There it's linear. I mean, you're just, and so the question is, how can you have your cake and eat it too? Enjoy the benefits of being to relate disparate kind of pieces of information and yet have something that's, if not linear, close to linear. And so Mamba, so first let's say Mamba is a straight, kind of left to right, what's called SSM model and the structured state space, but it's innovation was, it was a version that allows you to actually parallelize the training and much more efficient.

• 27:36Speaker 02

  But it still suffered from the lower quality of answers. And so what our guys did was say, okay, we'll take this as a basic building block. And Mamba is all of, what, four months old now? It's just cadaver recently.

• 27:50Daniel Whitenack?

  Yeah.

• 27:51Speaker 02

  But said, that's like a really good idea. But let's now take elements of the transformer architecture and put it in. So every few, in case it was every eight or 16, depending on which version, layers, you put an attention mechanism. So you take a little performance hit, but not nearly as much as if you had transformers all the way. So that's kind of how it led to this particular architecture.

• 28:16Daniel Whitenack?

  Well, Yoav, you did mention that Mamba is only a recently released architecture and published architecture, but you've been able to move quite quickly. And I want to talk a little bit about Jamba and the release and all of that. But prior to that, it might be interesting for listeners, you know, most of our listeners aren't sitting in a company that is trying to be a foundation model builder, building these kind of more general purpose models. I'm wondering if you could give a picture, a little bit behind the scenes, whatever you think would be interesting on what does it actually take to go from, hey, this idea, we want to mix, kind of get the best of both worlds with Mamba and transformers, all the

• 29:01Daniel Whitenack?

  way to, hey, here's our blog post, releasing a model. What were some of the challenges in that kind of middle zone? And what is that process like to determine, you know, from data set to exact architecture and the sort of final training runs?

• 29:18Speaker 02

  So first I'll say that I don't think that everybody needs to be building foundation models. But as I said to somebody, if somebody, a organization's a technical and wants to remain relevant, even if they're not building foundation models, they should understand how they're built. And if they really put their mind to it and their resources, they could build one. Because it really gives you a visceral, deep sense of what's going on. Now regarding the Jamba, we actually tried to be very transparent, you know, people. So this is our first open source model. And the reason we did it was that it is very novel.

• 30:01Speaker 02

  And there's lots of more experimentation to be done here. Optimization, serving the, uh, training these models can't be done on every, uh, type of infrastructure, serving them similarly. And where you do serve them right now, we've had several years to optimize the serving of, uh, transformers. We wanted to enable the community to innovate here. And so we were quite, uh, explicit in our white paper, perhaps unusually so relative to the industry. So to the listeners who wanted to kind of get the nitty gritty, I'm really encouraging to look at the technical white paper. But, um, I can tell you, there's been a ton of experimentation and ablations that our guys

• 30:44Speaker 02

  did, trading off very, lots of, people use the term hyper parameters. It's a, it hides a lot of things are very different from one another. But how many layers do you want? And, you know, how many Mamba layers, how many attention layers, batch sizes, um, all kinds of stuff that, and where, what really makes the difference? It's hard to sometimes understand what makes a difference. And again, we try to share the, for example, Mamba, I said that Mamba's performance doesn't compete with the performance of comparatively sized transformer models. But that's at the, when you look at the details, it's actually quite competitive on many of the benchmarks.

• 31:27Speaker 02

  But then there are a few that it's really bad at. And that gives you a clue of why that's the case. It can latch on to surface formulations and, and syntax that the transformers manage to just abstract away from. And so we describe how, you know, you made this observation, you correct for it. There's a lot of details that go into, uh, making these decisions. And then there's also pragmatic decisions. For example, we wanted a model that'll fit on a single 80 gigabyte GPU. That was a design decision. And from that emanated a few things that, you know, uh, we did put a bigger model and, you know, certain context windows will fit there, others won't.

• 32:09Speaker 02

  It's still, you know, 256K is humongous compared to the alternative.

• 32:15Speaker 02

  But we can also do a million and larger, but not on a single GPU. And so those are some of the design decisions and the rationale. Honestly, it's a process, although condensed, a process that involved, you know, hundreds of decisions that led to what we put out.

• 32:34Chris Benson?

  That was a really great explanation. I appreciate that. It's as you were going through it and I was thinking about the applicability for Jamba in the enterprise and kind of bringing the innovation. I'm curious is why, I know you had kind of alluded to the fact that Jamba early in the explanation was kind of the first open source model. And so I was wondering, as you're trying to enable enterprise innovation, what was the change in your thought process that made you decide to go open source with Jamba versus the earlier models? What was the thinking around that? I was curious, as you said it, and wanted to wait till we got to the end.

• 33:11Speaker 02

  Yeah, it really was very simple. We felt like if we were the only ones augmenting and pushing on this model, it wouldn't advance as fast as it could. And we saw that within days of our putting it out there, there was, I think today, I haven't tracked it, but when I looked about a week ago, the 30,000 downloads and I forget how many forks, but a large number of forks, some fine. So by the way, very important to say what we put out is a base model, not a fine tune model. And we're very clear about it and we caution people for using it for production purposes or for user facing application. And of course, we'll be coming out with our, in fact,

• 33:54Speaker 02

  we've announced that it's available for preview, our aligned model. But we felt like there's, it was really important for the community to add value to this architecture. And that's why we did it.

• 34:08Daniel Whitenack?

  For those that are listening a little bit later on the podcast, so it looks like Jamba, at the time we're recording this, was released, at least on Hugging Face. Well, it was updated 15 days ago, and I see the blog post at the end of March, I believe. But now on Hugging Face, there's sort of 38 models I see with Jamba in the name. That's sort of not including those maybe that forked and just created their own special name. So already you're seeing this kind of explosion of a model family, I guess, which is quite interesting. I'm wondering, over time, as a company, you mentioned kind of not being the only ones

• 34:51Daniel Whitenack?

  working on the model family and wanting to see it become more. Is that observation kind of based on what you've seen in other model families, whether it be Lama 2 or Mistral and others? And there's sort of, because when I look at a model like that that's released, I almost immediately, and I know people, you mentioned DevOps, people have automated pipelines in place to create the quantized version of this or fine tune it for that on their dataset. We had the Noose research, we had a discussion about Noose research and what they're doing in some of these areas as well. So what is the sort of innovation that you're hoping for with the kind of Jamba model family?

• 35:32Daniel Whitenack?

  Is it, you mentioned fine tunes, or you're releasing the base model, there could be fine tunes, but I think also there could be much more than that. So what are you kind of hoping to see as people get hands-on with the model and try to explore various elements of how to use it?

• 35:50Speaker 02

  Yeah, fine tuning is happening, will happen. Like I said, we have our own fine-tuned or aligned model, but that's not the reason we put it out there. The reason we put it out there is that people can contribute to the very model so others can benefit from it. And I think there's at least two areas where a lot of value can be brought. One is serving efficiency. For example, when you consume it on Hugging Face, it's less efficient than we consume it on our platform because we have optimized the serving and will continue to optimize. But a lot of smart people out there, and we'd love for them to optimize it further, and everybody will benefit, including us.

• 36:35Speaker 02

  That's one thing. The other thing is that I think if I really, we would really value it if this kind of model were able to be trained on multiple types of infrastructure, which currently isn't the case. And so I think by putting it out there, people now, they can look at the white paper, they can look at the model, and they can now enable further training of such models, which will benefit everybody, including us.

• 37:03Chris Benson?

  So as we start to wind up here, fascinating discussion. Thank you very much for taking us through all the insight. I like to wind up asking kind of where you think things are going, and if you could address it potentially at two levels, both kind of where your own organization expects to go, what kind of thinking you have over whatever horizon is on your mind, but also give us insight into how you think the industry as a whole is progressing and how you expect that kind of servicing the enterprise need to evolve and, you know, with the strategies that are out there. We'd love to understand how you're seeing the world in that way.

• 37:44Speaker 02

  I think the key notion is reliability, trust and reliability. You need to have the same kind of trust in the system to be able to predict what they'll do, be able to understand what they did, as you do with other pieces of software. You know, we always have errors, you know, even the Pentium had a bug, but that's an exception. Whereas currently it's the rule for language models. So that can't be in the enterprise. And everything that I think about what's going to happen in the enterprise orients around that. I think you'll see special purpose models like our task-specific models. I think you'll

• 38:26Speaker 02

  see AI systems increasingly sophisticated and robust. Right now, they're not robust, they're experimental, but you'll see more AI system. And I think this may sound philosophical, so bear with me. But there's a question within the AI community. Do these language models actually understand what they're talking about? They spit out this incredibly convincing stuff, very smart, something on point. And how can they not understand? And it's sometimes they're totally stupid. And everybody, we all have favorite examples. And I think we need to get to the point where we believe that the system actually understand what they're talking about.

• 39:10Speaker 02

  And what understanding is, is again, it sounds philosophical. And there's a philosophical aspect to it for sure. But it has very practical ramifications. And so when I think about the future, all these pragmatic things, task-specific models, AI systems. But in the background, this notion of understanding, these systems need to really understand. That's what I'm looking at.

• 39:35Daniel Whitenack?

  Yeah, that's great. Well, I think as a part of the development towards that certainly open models and innovation around these model families like we talked about, I hope, is a key piece of that. And from a member of the community, just want to express my thanks to AI 21 for being a leader, both in terms of the thinking and infrastructure and innovation in this area, but also a leader in terms of putting things out there for the community to work on as a community. So thank you for what you've done with Jamba and really excited to follow AI 21 and where you're headed next. So thank you so much for joining us, Yuav. It's been a pleasure.

• 40:17Speaker 02

  Thanks very much for having me.

• 40:26Speaker 00

  All right, that is Practical AI for this week. Subscribe now if you haven't already. Head to practicalai.fm for all the ways. And join our free Slack team where you can hang out with Daniel, Chris, and the entire changelog community. Sign up today at practicalai.fm/community. Thanks again to our partners at fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.