Sovereign AI in Poland: Language Adaptation, Local Control & Cost Advantages with Marek Kozlowski

Marek Kozlowski, Head of the AI Lab at Poland's National Information Processing Institute, discusses project PLLuM (Polish Large Language Models). PSA for AI builders: Interested in alignment, governance, or AI safety? Learn more about the MATS Summer 2026 Fellowship and submit your name to be notified when applications open: https://matsprogram.org/s26-tcr. He shares how countries like Poland can achieve AI sovereignty by training small, locally-adapted models for specific languages and cultures, ensuring control, privacy, and cost advantages. The conversation delves into challenges like frontier models' English bias, EU regulations, and technical strategies like "Language Adaptation" on base models. Discover how transparently created, locally-controlled AI offers a viable path for nations to maintain their technological destiny. LINKS: National Information Processing Institute Show notes source with images PLLuM open chat service Sponsors: Google AI Studio: Google AI Studio features a revamped coding experience to turn your ideas into reality faster than ever. Describe your app and Gemini will automatically wire up the right models and APIs for you at https://ai.studio/build Agents of Scale: Agents of Scale is a podcast from Zapier CEO Wade Foster, featuring conversations with C-suite leaders who are leading AI transformation. Subscribe to the show wherever you get your podcasts Framer: Framer is the all-in-one platform that unifies design, content management, and publishing on a single canvas, now enhanced with powerful AI features. Start creating for free and get a free month of Framer Pro with code COGNITIVE at https://framer.com/design Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) Sponsor: Google AI Studio (00:31) About the Episode (03:17) Sovereign AI in Poland (04:41) The Case for Localization (13:38) The PLUME Project's Mission (Part 1) (20:25) Sponsors: Agents of Scale | Framer (22:47) The PLUME Project's Mission (Part 2) (22:47) Defining Polish AI Values (35:32) Sourcing and Curating Data (Part 1) (35:38) Sponsors: Tasklet | Shopify (38:46) Sourcing and Curating Data (Part 2) (44:40) Small Models, Big Advantage (58:21) Training and Domain Adaptation (01:12:22) Compute, Talent, and Geopolitics (01:22:50) Forming International AI Alliances (01:27:41) Decentralized AI and Conclusion (01:31:47) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. While we often discuss sovereign AI in the Silicon Valley AI bubble, we rarely hear directly from the technical leaders who are actually leading national AI projects. And so today, I'm very glad to share my conversation with Merrick Kozlowski, who's leading Project Plum, which stands for Polish Large Language Models, in his role as head of the AI lab at the National Information Processing Institute of Poland. Poland, with a population of 38 million and GDP of roughly $1 trillion, roughly 10% and 3% of the United States, respectively, is an interesting and in some ways a representative case study. It clearly doesn't have the resources required to compete with the US and China at the AI

• 0:44Nathan Labenz?

  frontier, but it does have strong technical talent, a real sense of pride in its language and culture, and a deep desire to control its own technological destiny and avoid domination by global superpowers. So what does that mean in practice? As you'll hear, Merrick's strategy relies on the core belief that by training small models for a particular local language and cultural context, countries like Poland and projects like Plum can compete with the latest frontier models, all while retaining control, preserving data privacy, and achieving a major cost advantage. In this conversation, we dig into the strategic realities that motivate projects like Plum and the technical challenges that they have to overcome to succeed, including how today's

• 1:25Nathan Labenz?

  frontier models, which are trained on overwhelmingly English and Chinese data, fall short in other languages; why this problem is actually getting worse from one generation to the next as frontier model developers prioritize things like coding performance above support for niche languages; how EU regulation prevents European AI builders from conducting massive web scrapes and instead forces them to rely on more focused data curation projects; how the Polish government is thinking about investing its finite resources across data, compute, and talent; the language adaptation techniques that Merrick's team layers on top of Llama and Mistral base models so as to inject local knowledge without needing to start from scratch; why they haven't yet had to worry about developing a constitution or other explicit articulation

• 2:08Nathan Labenz?

  of values for Polish AI systems; and why government agencies and national champion companies are often better served by smaller models, fine-tuned for specific tasks, and served locally than by massive generalist models served from the cloud.

• 2:23Nathan Labenz?

  Overall, Marek's mix of realism about the challenges of competing with global leaders and his positive vision for transparently created, locally controlled AI is a great window into what AI leaders around the world are thinking and doing to maintain AI sovereignty. So with that, I hope you enjoy this deep dive into the meaning and training of Polish AI with Marek Kozlowski. Marek Kozlowski, head of the AI Lab at the National Information Processing Institute of Poland, welcome to the Cognitive Revolution.

• 2:54Speaker 01

  Welcome, everyone.

• 2:55Nathan Labenz?

  I'm excited for this conversation, too. We met not too long ago at an AI event in Las Vegas, the Enterprise Technology Leadership Summit, and I thought it was really interesting to double-click on everything that you're doing because in the United States and in the Silicon Valley AI circles that I spend most of my time in, there is this ongoing conversation about sovereign AI. And I think it's funny that a lot of this conversation happens in the Silicon Valley bubble and makes a bunch of assumptions about what other countries feel the need to have, aspire to create, what's driving those decisions.

• 3:38Nathan Labenz?

  And I don't hear too much from primary sources of people that are actually doing the sovereign AI projects around the world. So I was excited to meet you and learn more about what it is that you're doing in Poland. Poland, obviously, I think, obvious to me, is a country with a lot of technical skill and a very distinct culture, obviously its own language, proud tradition. And so I'm really interested to get into it and figure out what sovereign AI means in the context of Poland.

• 4:09Speaker 01

  Once again, thank you for the introduction and for the...

• 4:15Speaker 01

  For introducing my person and showing the idea, the idea is that I called it slightly broader, not only the sovereignty, but also the creating the localized LMS. Yeah, because the localized can be the national LMS, but also the domain oriented LMS. Yeah, and I create the, maybe not I create the idea, but I am, I am promoting the idea of the localized LMS. It means the LMS adapted to the language or domain, because they can be also a domain. And they are in this domain or the language, they have the, they have higher quality understanding texts in this language or domain, and have the higher quality and they are able to create the higher quality texts

• 5:00Speaker 01

  in a generation step. It means that building the localized LMS, they can be of course adapted to the language or domain, has two goals. First of all, to improve the understanding in this domain or the language, but, but also give the possibility to generate the higher quality texts, of course, in the, in the aspects like the linguistic and, and cultural aspects. They are, but they are the idea. And today, and our goal is to create the models that are the order of magnitude smaller than the close popular now LMS. But in the aspects of the language and the culture or the domain, they have the same quality as the 10 times bigger models.

• 5:42Speaker 01

  And they are open source, transparent, secure and, and as much organic as we can.

• 5:49Nathan Labenz?

  Yeah, okay, great, that's a great start. Can, maybe take one step back if we can, and just talk about like, why this is needed, first of all, from a capabilities perspective. Famously, I think it was...

• 6:07Nathan Labenz?

  Gosh, it's been a minute, but I think it was the GPT instruct series. I think the model originally was text Da Vinci oh oh two, if I recall correctly, one of the first models that OpenAI trained to follow instructions. They reported basically, we just trained this thing to follow instructions in English. And lo and behold, it seemed to be able to follow instructions in other languages, too, which was, you know, obviously a strong example of emergent capabilities and transfer learning, you know, positive generalization, all these sorts of phenomena that had been, you know, kind of elusive, but I think in many ways like characterize the the phase change that we've gone through from earlier AI systems to these more general AI systems,

• 6:51Nathan Labenz?

  positive transfer being obviously a huge one. But, you know, that's where they started in terms of like just English and, you know, oh, my God, it works in other languages. Since then, of course, they've gone and done a lot of work to try to collect data in other languages, to try to even things out. And my sense from just kind of benchmark data is that they have made pretty good progress, but still like performance is best in English. And then you can kind of think of like performance getting worse, sort of the farther a language is from English in the language tree. And also just the, you know, correspondingly, how many resources it has, right? Few low resource languages are obviously going to be a bigger challenge than than higher resource languages. That's my sense of like.

• 7:34Speaker 01

  Yeah, at least because I have to correspond to your insights. First of all, the 90% of the training data is English and Chinese. Yeah, when you even if you look at the biggest open source or the biggest closed LMS, 90% of 90% plus data are English and Chinese ones. Only 10% or less are the other languages. For example, it varied, but for example, in some models, the Polish language, I use this example, there is about 1% of the corpora or even smaller. And this decides that the the vast majority of the skills and the competencies are gained by the English and Chinese instructions. And of course, if you have the large model,

• 8:19Speaker 01

  it has a huge competency in transfer learning. You can say they extrapolate. They can very easily extrapolate between the tasks. But for example, even if I have the instructions, I have lots of mathematical calculations prompted or commanded by the English language. And even if I ask to do it in Spanish, the very large model can do it in the intermediate steps. It can translate on the fly. The commands from here from Spanish to English and somehow map the knowledge from the English to resolve the solutions, even if it was not learned on the Spanish examples of how to calculate some mathematical formulas. But what is the most important? That this works very good,

• 9:02Speaker 01

  but this works the same way as we or our kids learn the language. First of all, we learn how to understand and hear, listen. Next, how to write and how to speak. And of course, if you learn a new language and you get some commands in our minds, we try to map these commands to what we know from our the primary language, our native language. And the same is going inside the AI. For example, I can give you the example that the models that were not trained by the huge volume of Polish texts, they are still able to be communicative and create the text that is understandable. But there are some statements or some phrases

• 9:45Speaker 01

  that are very easily identified that they are not the natives. Right? For example, I give you the example about writing the emails in Polish. And for example, they have such a formula, "Difficult for English, I hope you stay in good health and condition." This is typical for English, but not typical for Polish. And even if you translate this word to word, it's communicative, understandable, but not typical for our language and culture.

• 10:13Nathan Labenz?

  So is that, is there more to say about how the leading commercial models are underserving the Polish market than that? I mean, I have, I have the sense that there is a little bit more to it than just the cultural idiosyncrasy. Because even when I look at like, you know, an MMLU benchmark, it does seem like performance degrades across the language spectrum, right? Like, the highest MMLU score is in English. It does seem to get worse in other.

• 10:44Speaker 01

  I know, but, but for example, when you look at the benchmarks, we live in the words that we are, we are biased by the benchmarks. It means, for example, the MMLU benchmark is mostly you, you choose the solutions A, B, C, D. Yeah, the multiple choice questions. They are not testing the ability to communicate fluently in the language. Yeah, most of the benchmarks don't test how good is the model in producing the longer forms, longer writings or longer sentences. Yeah, they usually test the understanding, extractive competences, summarizing competences and, and many of knowledge about the facts in the world.

• 11:24Speaker 01

  But there are, I know, a little or almost little, very little, very few benchmarks that test how good the model is in generating the longer forms of text in the other languages than English and Chinese, for example, the niche languages, yeah? Because it's much harder. For example, we in Poland create the benchmark PLCC, Polish Nationalistic and Cultural Competency Benchmark. And this benchmark enables us to evaluate how good is the model in different subcategories. And for example, they are not only categories of the grammar vocabulary, but also about our culture, tradition and history and many, many others. But we would like to not only evaluate how good is the wordings of the model,

• 12:06Speaker 01

  how good the model is in some typical for our tradition wordings and the phrases about the history also, but, but, but we also try, trying to check how good the models are in the general spectrum of the using some, some ambiguous words for the, in Poland. But there is still, but there is still not, this benchmark still doesn't validate, validate how good the longer sentences are in Polish, how good the model is in producing the longer structures in, in Polish language.

• 12:38Nathan Labenz?

  Yeah, interesting. So is it fair to say that the primary focus of your work in creating Polish native models is on this, on these sort of softer skills? Like, it doesn't sound like you're focused on closing the benchmark gap, or the sort of reasoning gap that exists between English and Polish. It's more about, as you said, like, culture values, tradition, history, cultural competence.

• 13:07Speaker 01

  But this, because I think that the language is not only the wordings, you know, the models can have the very broad vocabulary, but they should be able to use it properly in the context. And sometimes the language is not only the words, there are the culture, tradition, history, everything is mixed in it. And in order to create the model that behaves as natives, you have to inject not only the knowledge about how to create grammarly correct sentences, but also how to use special idioms or phrases in special contexts, or what places are typical for Polish history, or maybe what places are viral now. Generally, it's like

• 13:51Speaker 01

  where you have to mix the history, grammar, vocabulary, art, entertainment, culture, and tradition, everything into the mix to create the language ability that is somehow similar to the natives. But ask me why we are doing that. First of all, because as I mentioned, sometimes that we believe in the idea of the localization elements, the localized elements, because the elements adapted to the language, that they are as much similar to natives as possible. The second issue that the competency gap, that for example, we believe that, for example, we have to develop our people, our engineers, to have the skills to beat our

• 14:33Speaker 01

  own models, because maybe in a few years, the market will change, maybe the mobile will be closed, or maybe some models will be forbidden. There are plenty of models currently in the European Union, but they are not, we are not able to use it because of the IA. Even in the licenses, the Lama 3.4, the Kimi, and many, many other models, they have the sentence, the statement in their license that they are prohibited to use in the European Union. Maybe we will be forced to use this knowledge to beat our own models. Maybe they will be a little bit worse than the Chinese or USA, but they will be our own. Yeah, and sometimes it's better to have the competences to beat even something a little bit worse, but have the ability to do it than don't have this.

• 15:18Speaker 01

  Yeah, we can, sometimes it means more than you think. Uh, but also we think that there is in this approach, in the Plume family, because we create the family of models, we also believe in the transparency. Because we show the how we built it from the scratch. We we we released a few weeks ago, or two weeks ago, sorry. We released the publication about almost one hundred pages, how we build these models. Yeah, and also we not only released the publication, the the the recipe book, uh, the cookbook, but also we published on the Hugging Face the samples of our datasets, the instructions, preferences, because we would like to show not only the open weights,

• 16:02Speaker 01

  but the more and more than, because the open source are not only the open weights. There are also there's some samples of open data and the cookbook, how we do it step by step in a very detailed manner. And what is for us important, because I think even now they are the most popular open source models are the Chinese now, but they are only open weight. There are no samples of the instructions or preferences they used to train the models. And here we would like to go step farther, to to be as transparent as possible. And also we invest lots of in their organic or genetic data, because we we believe that, we we also prove that, that when you uh, you have three stages on the, you know, when you learn the models. First is the pre-training.

• 16:47Speaker 01

  It's somehow similar to learning the kids a new language, that you you identify the words, how to create structures from these words and some pieces of information. But children after this type of learning is not able to resolve the mathematical calculations or write the essay. Yeah, it's like you learn the language, but you don't learn the competences. Next stage is the SFT supervised fine-tuning. You learn how to resolve some tasks, downstream tasks, right? The poem, write the essay, summarize this article, perform some calculations. You learn the competences, like the children in the school. You have the math, the geography, chemistry and many others.

• 17:29Speaker 01

  And after all, you have the alignment, the preference learning, that you mark what the children have done during the test, for example. And this information, these marks, makes them what should be corrected or not. Yeah, and with the same, what you have done with the children, with the kids, that we learned the language, the piece of information, the wordings and the structure. Next, we learn the competences and evaluate them. And during the feedback loop, we try to improve their abilities. The same things are done with the LLMs. And for example, when you're doing the pre-training, you show the model the hundreds of billions of tokens to learn the language. And after all, in the SFT, the supervised fine-tuning stage,

• 18:13Speaker 01

  we show the model the synthetic instruction. Synthetic means that they are produced by the other LLMs. They usually, if they are linguistically poor, they also degrade the model. Yeah, because if you, in any stage of the learning, when the model sees the poorer data, it will degrade, it means it's going down, there is quality of the linguistic creation of the sentences. And so we focus mainly on the create the organic data sets, organic instructions and preferences, or even if you use the LLMs to produce such instructions, we check it by the humans to improve their structure and quality.

• 18:58Speaker 01

  And I think they are the novelty in the and the second novelty is that the first of all, there is the open source, open, open data and open cookbook. The second is transparent because it's written in the cookbook, what we have done step by step, and we showed the samples. We also focus on the organic data, organic instructions, organic preferences. And I think this is the reason why the GPTs and other people are so good. Because they are also, they have plenty of manual instructions and they don't show them because there is an intellectual property of these companies. And we also, we secure our models on our own because we discovered that these models, when they are secured for the English speakers, they can be much easier hacked than the secured

• 19:43Speaker 01

  for the Polish speakers. I think they are the novelties, maybe briefly speaking, yeah.

• 19:51Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 19:56Nathan Labenz?

  I have like seven follow up questions I want to ask about various parts of that. And maybe we can kind of break it down by inputs to AI for one thing, right? Obviously, the big inputs are data, compute and talent. And you touched on certainly data and talent there. I also do want to come back to the, the safety training because that's always a keen interest of mine. But maybe let's start with like the goal. You've spoken about it somewhat, but I think one big challenge that we have, certainly in the United States, and we have all this talk, right, of especially in the context

• 20:35Nathan Labenz?

  of the geopolitical competition in AI, there's a lot of talk about, well, we want to have AI with democratic values when, you know, we don't want to have Chinese values or maybe we even are bold enough to say we want American values to be the values that the AIs. You know, embody and and kind of propagate through the world. That obviously brings a big question, which is, well, what are those American values? And I can certainly say that there's no like single agreed upon answer for that. Right? What American values are is like hotly contested on an ongoing basis. And that leaves basically the AI companies to try to come up with their own, you know, best guess of what that should be.

• 21:18Nathan Labenz?

  And, you know, that that too is like often sharply criticized because, you know, it's too woke or it's, you know, not woke enough or it's, you know, right wing extreme or it's, you know, it's it's describing itself as Hitler.

• 21:29Speaker 01

  In some of the L L M's, they are somehow they are the compressed representation of what we have in our web in the internet. Yeah, there's somehow what topics are the most important, what our information is the most popular ones. It's somehow is reflected by the L L M's. Yeah. If you have the the problems, the political problems, the religious problems, the everything what is there is also reflected somehow in the compressed L L M's because L L M's, they are somehow they are compressed memo memories repositories. Yeah, they are the compressed stores of the memory of the internet.

• 22:08Nathan Labenz?

  Certainly, that's, you know, all that stuff is baked in. Sometimes I don't know how far American, the leading companies have come today in terms of filtering the training data. I know that there are some techniques that are like, we're going to get rid of all the bad pre-training data, you know, and just try to show.

• 22:27Speaker 01

  Yeah, we there is a there is a typical step. Yeah, even even we in our project, there are this called data curation that even because the as I mentioned, in the pretending stage, the 90% of the data are the web data and the web data are plenty of them is creepy. Yeah, you are not able to use them because the model will be not stable. And in this this data creation step is there are two stage, sub-stages that the duplications, you remove the same information that is repeated very often in the internet. It's sometimes it's it's scaled to two times your because there are, because there's plenty of duplicates duplicates in the internet. And also there is the filtering out. We filter out the data that are very crappy because they are low quality.

• 23:09Speaker 01

  It is, for example, there are plenty of special characters, plenty of interfunctions. Plenty of not words from not not being recorded in Arabic vocabulary. Yeah, there are plenty of such a such a disturbed data. It should be that it should be truncated because it will have impact on the stability of the quality of of models.

• 23:32Speaker 01

  And I think you mentioned that that the big companies, they have the such a tools, but are not even able to eliminate some poor quality data. But they are even eliminate some, for example, the theories, some points of views and and and much, much broader section, not only the linguistic aspects of the data.

• 23:54Speaker 01

  So it's the same censorship. The Chinese models, if you ask about the what has happened in Tiananmen Square, they are not not able to give you any information. Yeah, they generally, the the people who built models, they can able to isolate or how is called the black the band some part of informations that for people who heard about that will be the reflection of the words we told some part of it.

• 24:24Nathan Labenz?

  Yeah.

• 24:26Nathan Labenz?

  But there there's at least like two layers to this, right? I mean, there is all this data pre-trained or pre-filtering. And I genuinely, I could certainly believe that the Chinese models are trained on data that's so thoroughly filtered as to never have seen, you know, any document about Tiananmen.

• 24:41Speaker 01

  I I think that even not about in the pre-training stage, I think in the last stage, because as I mentioned, there are three, three stages during the learning of models, the pre-training, the SFP supervised training and the preference learning, sometimes called the reinforcement learning with the human feedback, but there are other methods like DPO or and I think in this stage, they secure their model, how to not behave.

• 25:03Nathan Labenz?

  Right. So that's what I want to get at for what you're doing in the Polish context, because in, I don't know what, you know, the Chinese companies are doing. I do know that the American companies are developing their model specs or their constitution.

• 25:18Nathan Labenz?

  You know, it's basically this super long document that says, this is how we want our AI to behave. And to their credit, you know, they're starting to be reasonably transparent about what those are, so that at least the public has a sense of like what they're going for. But again, it's like it's in the US context, it's like pretty contentious because, you know, everything is is contested here. In the Polish context, is it like that? Or is it, you know, is it an easier time? Like, do you have a constitution for what you want Polish AI to be?

• 25:49Speaker 01

  There are some strategies. Like, there were the strategy, how the AI should behave. Or no, maybe not. How should not behave. Maybe in this case, there, for example, there should be ethical, that should not blame anyone, or for example, to avoid some topics that are, for example, very risky topics. Yeah, they have to say the topics about the hate speech. Yeah, there are some places that there are typical, especially for the models from the Chinese or the USA, there are the place when there is a risk that the model behaves in unethical way, or in a way that we can, that can be blamed, that is not, no, it's not the, it's not, it's rude, at least, at least rude.

• 26:31Speaker 01

  But of course, there we go, some political tensions, yeah. But generally, I think we don't have such a huge, now currently, very huge constraints, as you mentioned. Yeah, we don't have a constitution that has plenty of points that you have to obey. I feel we are mostly that the models should be ethical as much as we can, but we don't give too many constraints to the model, because I think we are on the other level of development of China, Chinese or the USA government or the companies, because we are, I think, a few years behind them, or two years or three years, it's hard to say. But generally, I think we are not, don't have such a, we have our own regulations, but

• 27:15Speaker 01

  not the regulations containing the, or how the model should behave, but rather what kind of data we are able to use for training. We have many constraints, type of like that, focus on the data, then how the model should behave.

• 27:28Nathan Labenz?

  Yeah, interesting. So, I mean, I've never even been to Poland, so obviously, you know, should be very humble in terms of my ability to describe it, but one high-level fact that I know is that the large majority of Polish people identify as Catholic.

• 27:46Speaker 01

  Yeah, it seems to be. For many years, it used to be a good statement. Currently, I think it depends on the, on how the city is big, that the inhabitants of the cities versus inhabitants in other villages, I think they should be varied.

• 28:02Nathan Labenz?

  Yeah. So how do you think about that dimension? You know, should the, I just happened to have done an episode not long ago about Catholic AI with a company that is literally building AI that embodies Catholic values, you know, specifically for religious Catholics. But in your context, you know, you've got this sense that like, okay, well, you know, maybe a majority of people are Catholic, but maybe that's on the decline. And maybe it depends on, you know, an urban rural divide. Do you have, is there some sort of decision-making process where you think, okay, like, how Catholic should our Polish AI be? And, and, you know, does it vary in different situations? Is that, are you guys getting explicit about?

• 28:44Speaker 01

  I feel

• 28:44Nathan Labenz?

  articulating goals there?

• 28:45Speaker 01

  I think we are much more liberty now. I think there is now, now we don't have such ideas to create the reflection of our world. But as I mentioned, there, I think there is very, it can be very easily, the models can be very easily constrained by the, by the preference learning. And you can learn it to behave in such a special way. But now currently, when we produce the family of the models, we produce not only the chat models, but also the instruct models, the base models. We give the possibility to companies to use any kind of the models because we know that some constraints may have a disruption effect on the some business cases. But generally, I don't think that we, as the producer of the, of the, as the researcher, or the, no, the, the builder of the models,

• 29:27Speaker 01

  we should get all of them, the people, and people we, we decides how to skew them.

• 29:35Nathan Labenz?

  Gotcha. Okay.

• 29:39Nathan Labenz?

  Yeah, very interesting. Do you, do you envision that this will become something, you know, as, as you presumably go on to train more in future models and they become even more powerful and, you know, potentially?

• 29:56Nathan Labenz?

  I don't know to what degree you sort of aspire to serve a consumer use case versus, you know, empowering businesses in the country. But do you think that this becomes a challenge at some point? Do you envision a future where there is a sort of Polish constitution for AI that actually seeks to answer that question? And if not, like, how do you think you ultimately get around that? Because it seems to be a very central thing that the American companies feel they need to grapple with. So if you think you can avoid that problem indefinitely, I'm kind of wondering.

• 30:27Speaker 01

  I think we have much harder problems because, for example, you have the AI constitutions in the companies and in the USA market. But for example, you can very easily use all data you have without any constraints. Of course, there is a problem with some suits and many other cases, but there is like, maybe it's a long process, not very, I think for most of the companies in USA, they can take this risk because they are still profitable enough to even to pay some, how to say, some bad decisions by the judges and so on. I have to say the arbitrary decisions. But generally, I think in Poland and European Union, we have the AI Act and our local regulations,

• 31:11Speaker 01

  like for example, the Acts concerning the, how it's called, the rights, the ownership rights. And fair, I think these problems, legally speaking, these documents, they have much harder impact on our quality of models than we can, I think it's enough, it's enough huge constraint not to go further, yeah? Because as I mentioned, in the European Union, you have the AI Act concerning the general purposes. We have also our local regulators, like for example, the AI Act about the, the Act about the ownership rights. And both of them combined create a much harder constraints than, for example, any kind of

• 31:52Speaker 01

  constitution that is much, I think, flexible, more flexible than our regulations. And I think we don't think currently about the AI constitution, but I know that maybe in one, two years, I think that, I think like that a few. But currently in European Union and Poland, we fight with the currently existing and obeying the regulations.

• 32:12Nathan Labenz?

  Yeah.

• 32:13Speaker 01

  And I think they are much harder, much harder, and they are much impactful than those you mentioned in the USA. Because for example, you have the AI Act or the Autorship Rights Regulations, for example, they can eliminate the eighty percent of the data from your training training data sets. And so it's a huge, they have a huge, it has a huge impact on the quality of of models.

• 32:41Nathan Labenz?

  Yeah, okay, that's interesting.

• 32:43Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 32:47Nathan Labenz?

  So turning to data then, we can check back in on the state of the Polish AI constitution in a year. On the data front, you had mentioned that in the biggest open source models, maybe one percent of the data is Polish. You know, quick back of the envelope math, I think the, you know, the Llama models are like maybe up to fifteen trillion tokens that they've been trained on. I don't know if they disclose their data mix, but you know, that would cash out to something roughly on the order of a hundred billion tokens in Polish that the biggest projects might be using. I understand you have quite a bit more data than that, but also your last comment.

• 33:26Speaker 01

  We don't have the trillion of tokens, yeah, because even as I mentioned, even the Llama, there was about, as I mentioned, the one percent is the Polish language or maybe less. We have now the several hundred billions tokens. We don't have even the trillion of tokens. Because as I mentioned, the duplication stage, the duplication stage and the free tearing out stage, they eliminate lots of data. And we don't have one trillion tokens after these data creation steps.

• 33:55Nathan Labenz?

  So where are you getting your data? And your comment about like? The difference, the sort of regulatory arbitrage that the American companies are potentially taking advantage of, are they able to use some Polish data that is on the internet? But not.

• 34:12Speaker 01

  Because I remember, I remember that it was some times ago, there was the people who analyzed the crawlers that are going on the websites in the Polish, in Poland. They identified there are plenty of anthropic crawlers and there was plenty of robot takes the pages on the websites that they disallow these anthropic colors to get data. I mean, it means there are plenty of the colors from the US companies, researching companies that are crawling the Polish data, even if they are not, if they are, even if they are not allowed to do it. Because as I mentioned, it's much harder, for example, to be there, I know, to have the court in the USA and to have some accuse them for using the data and able to fight

• 34:57Speaker 01

  with them in the USA court. Even if you have some proofs that they use data, that they have the disallow clauses.

• 35:06Nathan Labenz?

  And so in that way, you're, if there's 100 billion tokens that they're getting off the internet, it sounds like you can only use a fraction of that. And then you had to go elsewhere to find the hundred, a few hundred billion tokens that you've.

• 35:22Speaker 01

  Yeah, so they're using the central libraries with some sources of data that are not web. Yeah, because as I mentioned, the vast majority of data used by the big vendors, also by us, my web data. But for example, you have also some data that are not published in the web and you can use to somehow, of course, to some extent, use them. But as I mentioned, there is the minority of data. Still, even for us, even if you have some access to the local organizations and so on, the vast majority of data we use, there is web data. And the problem is the same for all other players. Yeah, maybe we can much easier identify some websites that are not easily crawled by

• 36:07Speaker 01

  the external colors. But generally, I think that most of the companies, the OpenAI or the Anthropic, they have, I don't know, maybe 80 or 90% of our data still.

• 36:18Nathan Labenz?

  Hmm, yeah. So where else are you going to get data? Like, what is your data process?

• 36:24Speaker 01

  We have, for example, the first of mass machine data, but also we have, for example, the, the, there is called, how is called the, uh, the library of the science, yeah? The, there is a plenty of publications and so on. We also have done some private bilateral agreements with the, with the publishers, but not published on the web. But as I mentioned, it's only the fraction of data we have in our corporate.

• 36:55Nathan Labenz?

  And are you also, you mentioned kind of doing a lot of human review.

• 36:58Speaker 01

  Yeah, is there? There are the huge, the, the, what we have, well, our advantage is not in the data used for the pre-training stage, because as I mentioned, I think the 80% of them is still in the anthropical open area repositories, but that we have dozens or even the hundreds of our titles that create the manualized instruction preferences. Yeah, because they give us the ability to create the new data that is not published in the, on the internet still.

• 37:29Nathan Labenz?

  Is there like a Polish equivalent of like Scale AI or Labelbox that you're working with to do this? Or is this a project?

• 37:37Speaker 01

  No, we have made our internal tools, not, not, not so crowdsourcing ones.

• 37:44Nathan Labenz?

  So you guys have built your own platform for human preference?

• 37:47Speaker 01

  Yeah, we can preferences and the human instructions are built locally internally. Of course, some of them, some samples of them we publish to show what is the structure of our instruction preferences and some examples, but most of them is still the closed asset.

• 38:05Nathan Labenz?

  So how do you think about that? I mean, one question that I've been thinking about in the context of this whole sovereign AI discourse is

• 38:14Nathan Labenz?

  Obviously, you know, as a national government, you can have different strategies or different goals and different strategies for what you're trying to do. One goal is, as you alluded to, we want to make sure we have our own base, you know, our own data, our own talent base, compute, we'll get to in a minute, so that if we get cut off, or, you know, who knows what might happen, we have some sovereignty over, you know, what's going on, right? I think that's

• 38:42Speaker 01

  Yeah, some possibilities to develop in a different way, yeah. But also there is the second, because this is the one, as I mentioned, we can something, we can mean something more than you think, yeah. If you can do something, even a little bit worse, there is still the competency and possibility to do a new way, it's a new movement. But I think there's also the second, the second issue, that I believe that the AI agentic revolution will be based on the smart localized models. Why? Because first of all, there are some branches or sectors in the economy, or even in the, in our public sector, that we are not allowed to use the cloud-based solutions.

• 39:25Speaker 01

  Yeah, there are some regulations, or the, we are not, the risk is too high, and there are some demands to have the on-premise, on-premise models. When you have the premise models, you always have some challenges, like for example, the GPUs you have to buy, the energy consumptions, and usually when you realize, for example, you need to buy the 16 GPUs and the, and the pace for energy, you always go through the downscaling, yeah, to, to use the as small model as possible to achieve the expected goal. And from our experience, that for example, the people, and especially the businesses, but also the, the public sector, they don't demand exactly the chat GPT.

• 40:10Speaker 01

  The general purpose L L M that is able to resolve one thousand tasks. Usually in the business and the public sector, you have the demands for the ten or twenty use cases. And you are able to create the smaller models that are able to resolve this task on the same level as the few shot use big L L Ms and hosting them on the previous solutions. And I think when you go to the AI agent solutions, there are plenty of agents. It means there are plenty of models that are used to resolve some complex scenarios. You really have to downscale the models to using only the as small as possible models to be energy effective and also the the economy aspects are crucial now.

• 40:52Speaker 01

  And I think this is the place where the small localized models are can play well.

• 40:58Nathan Labenz?

  Yeah, that makes a lot of sense in the business context. And yeah, you know, in my experience, I would say the same has been true. Like when I'm really trying to dial in performance for a particular use case, and that's all I care about. And I kind of know that this model is going to be deployed in a controlled environment where, you know, because of the way the system is set up, I know what the inputs are going to be. I know what the outputs are going to be. I know that, you know, I have a I have other layers of control. Then, yeah, I can just kind of dial into one task or a few tasks. And often a cheaper model, you know, with the right training can do just as well.

• 41:37Speaker 01

  Especially when you have the because mostly the people are now using the the cloud L M's in a few shot manner, yeah, because they are very powerful. They are able to resolve the very broad number of tasks, thousands of tasks. And they use them as the in a zero or future scenario. It means they are integrating the A P I of their of their own systems with the cloud based L M's. And they use them as they can out of the box easily. Because you could create the prompt and use the output, that's all. But when you have the when you need to create the much more controllable solution, my the close solution within the previous solutions. You're not able to use cloud solutions. You have to go with the different, as I mentioned, the aspects of the decisions.

• 42:23Speaker 01

  Do I need the multimodal or textual only model? Do I have a training data sets? Maybe if I have any data sets, I can supervise, fine tune the smaller model and achieve the same quality as the future applied cloud-based solutions. And mostly, we can, we create many deployments currently. We identify that when you have the few or ten different use cases and you create for them at least 1,000 instructions or 3,000 instructions and you SFT supervise, fine tune the smaller models, you achieve almost the same quality or maybe sometimes a higher quality than using in a huge shot approach the big, very big large cloud-based LLMs.

• 43:05Speaker 01

  But of course, you have to prepare training data sets, at least 1,000 instructions. The best, the higher number of instructions is better. But 1,000 is enough. It mostly organic ones, but or maybe the semi automatically created with the human factor. And then you have 1,000 or more of the instructions. You can SFT smaller models. And in this one task or a few tasks, you will have the same quality as using the zero or future cloud-based Sams. And I think this is the the future. Because I think if they, if they we got the business and the business will calculate the risks, the money, the possibilities, the how they control the solutions, how, what is the impact of their decisions?

• 43:47Speaker 01

  They finally, in this AI agentic environment, choose the small local models and fine tune them to their demands. And there is also one risk, I will show you there, because we discussed it last time, lastly with the Saliglabas and of my colleagues. But for example, in the Atropics models, the the cloth, for example, the the LLMs from the Atropics, they, for example, their quality in the ability in in knowledge about the Polish language and cultures going down. Because, for example, they decide to be focused mostly more on the software developer assistance. And if they focus on the this might have a point of fraction of the market,

• 44:27Speaker 01

  they go with the down with the qualities in the, for example, generating texts in the niche languages. And for example, imagine that, for example, in Poland, you apply such a models from Atropik. You integrate it with your environment, with your ecosystem. And after the next months or next years, the next releases of this model going down on your competencies that you demand. And you have to choose the other model, or you roll back if you can. Because unless you are able to roll back, revert the previous models, or you have to choose another vendor. I think the creating the huge integrations based on the cloud L M's.

• 45:09Speaker 01

  It's a huge reason because, as I mentioned, during the next years, they can change their target objectives and don't need to focus on the Polish or Czech Republic languages because they are not the market for them.

• 45:22Nathan Labenz?

  Yeah, that's really interesting. I've never heard that before. So just to make sure I understood correctly. You are seeing worsening performance over time in Polish on like top issues of sort of Polish culture, general world knowledge as the cloud models have progressed through generations.

• 45:42Speaker 01

  Yeah, because we have the, I can say there is the anthropic based models, I think the cloud and then the Haiku, or maybe I think the Haiku. But also we identified this problem with G P T models. For example, the G P T models that don't going up with the equality of the Polish language and culture competences, there are some of them even going down with the next releases. I think the problem is much more broader. I think that the creators of the models, they analyze the market. And for example, if they need to focus on some competences and they improve these competences, the sometimes there is a trade off that other competences are going down. For example, in this case, the Polish cultural and linguistic competences.

• 46:26Speaker 01

  I can check it. It was the cloud model for anthropic and this version is going down on this our P S S C benchmark when you compare to the previous releases.

• 46:36Nathan Labenz?

  Wow. Okay, that's a really interesting data point.

• 46:43Nathan Labenz?

  Yeah, and I guess it maybe

• 46:47Nathan Labenz?

  sort of answers the next question that I had for you, which is my general working model has been that the AI frontier model developers want as much data as they can get, you know, and if you had any data for them, they would be happy to take it and maybe even pay you for it. What you're saying sort of suggests, well, maybe not always because, you know, they're trying to do the smallest models that they can as well while, you know, they're doing all this sort of distillation. They're trying to, you know, go for efficiency. They're trying to obviously serve the core use cases that they're getting paid for, which is, you know, a lot of coding. And so maybe if you showed up at their doorstep with

• 47:27Nathan Labenz?

  a few hundred billion tokens worth of Polish data and said, hey, would you like to use this? Maybe they at this point might say, not really, because we aren't that focused on that use case versus we'd rather, you know, go do another however many billion, you know, generations of token of coding tasks and and use those tokens instead. I guess, would you guys ever consider? I know you have some, it sounds like you have some open data, but not all of it is is open. If I was the, if I'm thinking as the government, I guess another goal that I might have is I want my users, retail users, you know, just my general public to be as well served by AI as possible.

• 48:07Nathan Labenz?

  And I don't know if you have stats on like what the Polish, you know, retail consumer is using right now. Are they going to chat GPT? Are they going Gemini or something else? Mr. All who knows? You may know, I don't know. But if I was the Polish government and I was saying, okay, my, here's what my people are doing. They're using these other companies. We've gone and collected all this data. Is there some sort of deal to be made with the AI companies where you might say, hey, we'll either give you this data or perhaps license you this data. You pay us for it and that way you can incorporate it into your process. And that way you can serve the Polish market better. It's I've wondered if there's a safe trade that.

• 48:46Speaker 01

  It's a, it's a good point of view that is, I think the next natural step, yeah, that you are not able to get more data with some assistance or the cooperation of other players. What does he mention about the previous statement about the to serve as good as we can for the our citizens? And I think the this is still the this one of the goals of our projects. Because about the plum, the family of models that they call plum, which language models is the family of models, but not only the family of models. But also the assistance and chatbots for citizens and city inhabitants. Yeah, the video not only the focus on the models as itself, because the models are very

• 49:25Speaker 01

  good asset, but how to build based on those models, the chatbots, the lack approach solutions that can work for the citizens nationwide, but also in the for city inhabitants for the for the local chatbots, for example, in the cities, of course. And as I mentioned, it was the it's very good to to mention that, that sometimes we are trying to create better and better models, but the problem is for example on the digital level and then other other another other place. For example, there is not a problem that the model is a little bit worse or better, but for example, there is no chatbots for the cities, municipal halls and so on. Yeah, there is a two issues.

• 50:09Speaker 01

  First of all, the deployment issue, this model should be a little bit customized, supervised, fine tuned, to be able to work as the part of the of the citizens and and and and bots and assistants for the for the city inhabitants.

• 50:25Speaker 01

  And this is the one issue. And the second issue, as I mentioned, that is sometimes maybe there is time for some corporations, because the only the corporation gives you the ability to improve your data sets and improve your models. I think it's a very good step and I think, as I mentioned, it's it's we are we are developing in our pace, but we know that there is a place where you are not able to go further and you have to be supported by someone else. Yeah, it's normal in each, the same is in the business, yeah? To some level of your development, you are you you you reach some points, but you have to be supported by some better players or diverse players to be better finally.

• 51:06Nathan Labenz?

  Yeah. So do you know what that kind of market share breakdown is today? And is there a sort of established goal that you have to win market share with the models? Or is it?

• 51:21Speaker 01

  It's very hard, certainly, about the winning. And I think because we are the, the blue models are not the corporate initiative. It's not the, there is not the private money and the funds. There is the, this is the project supported by and funded by the Ministry of Digital Affairs. It's a consortium of the six institutes and now the eight, because we emerged in the second year, institutes and universities. And we are the public initiative. If you are the public initiative, you don't think too much about the, about the return on investment or the number of customers. We are much more focused on the how to be the as open as possible, how legal, because you have to be legal compliance

• 52:03Speaker 01

  with two regulatory holders, transparent as possible, organic, because it proves there are linguistic possibilities and secure. And how to can be used by the public sector as much as we can, because mostly in the public sector, the models can be, should be close on on your previous deployments.

• 52:22Nathan Labenz?

  Yeah, gotcha.

• 52:26Nathan Labenz?

  How, you've shared a lot about how you train these models, but what is the base? You're not doing all the pre-training from scratch, right? And my understanding is.

• 52:34Speaker 01

  Oh, starting with the base? I can say about it more, I can say it and explain it much more in detail. We're trying to create the models from the scratch, it's from the random weights. But the problem is in the pre-training stage, the number of tokens you have. As I mentioned, even there, I, when you look at the April reports from the DSLM, DSLM models, they prove that even if you have the eight billion parameters models, you have at least one trillion tokens to have the stable training, it's stable that gives you the moderate or high quality base model. And in our case, as I mentioned, the duplication stages and filtering out give us around 200 billion tokens, just too little

• 53:17Speaker 01

  to create the model from the scratch. And we use the, of course, the Lama. Of course, now the Lama are much more closed, but one year ago, they were still open and the Lama license was not so prohibited in European Union as it is now. At least now, but usually we use the Lama-based models and the Mistra-based models. And we continue pre-training. We perform the language adaptation. Language adaptation means we continue pre-training them on our corpora of Polish texts. And after all, we have the new base model. And this new model, new base model, has the SFT and different optimization learning in the next second and third stage.

• 53:56Speaker 01

  But as you mentioned, we are not able to create the, as I said, the moderate quality or the good enough quality model without 1 trillion tokens, and we don't have 1 trillion tokens in Polish language. Now we have made some experiments with the mixture of languages. Yeah, we may not only the Polish language, but we also use the other languages. The mixture of language to get this 1 trillion tokens and to make some random pre-training from the scratch. But the address will be in a few weeks.

• 54:26Nathan Labenz?

  Hmm, okay. On this language adaptation step,

• 54:32Nathan Labenz?

  I have a couple of questions there. One is, do you continue to mix in English? Do you try to preserve the model's ability to speak English? Or after this language adaptation, does it just only speak Polish?

• 54:43Speaker 01

  Of course, there is, when you have the cascade learning, because if you use the base model that was already created and you perform the language adaptation, of course, there is always the cascade learning, the problem of the forgetting. Some knowledge is forgotten from the previous learning stages. But there is still some persistence still now. Even if we pre-train a few epochs on our Polish data, the models still have the competences, for example, to write Sapikian English. There is not, we don't prune, we don't prune the, any other competencies on the other languages. They have, they somehow they are forgotten, because there is a problem of forgetting in the cascade learning. But generally speaking, we don't prune it manually.

• 55:28Speaker 01

  You only take, take the base model and perform the language adaptation, continue pre-training a few epochs. And in this how that we makes better the abilities in the Polish language. Of course, there is a trade-off. The some other languages are going down, but they are not pruned at all.

• 55:46Nathan Labenz?

  And that's also where the world knowledge comes from, right? And by world knowledge, obviously, there's, you know, all these sort of local details of life, right? And I, you can maybe give better examples. I'm sure you can give better examples than what I would give, but I'm thinking just like, you know, what are the names of the Polish candies that kids like? And, you know, how does one file a document if you want to sell a car to somebody else? There's, you know, surely some filing process. All these sort of little details, they're absorbed in that stage as well, right?

• 56:15Speaker 01

  Yeah, and you have to know that, that usually when you, the knowledge, the general knowledge is pertained. But then you ask me about the factuality, how model is good in some facts, for example, regulatory issues, some law issues that are changes in the time, they are changing in the time. It's always the problems in any kind of L M. Yeah, because you pre-trained the L M, for example, on the data that are till the March 2025, and you don't have in this memory store, information about the, about the changes in the law or the, some regulations or even some situations, accidents and the names of the new politics after this time point. But generally, we use it, even use in the RAC approaches or data.

• 56:59Speaker 01

  It means you have this retriever stage, when you have the database, the knowledge base up to date, it's much easier to be updated. And after all, when you get the some retriever stage, you use the models to synthesize or generate the answer based on that. And we, in this case, we, we use this factuality issue, because I think none of the providers of the L M, even the big ones, are not able to pre-train it any new interval of data is coming up.

• 57:25Nathan Labenz?

  Yeah. Do you think this would work for companies? This has been sort of a, this is a bit of a digression, but it's been a question I've had in my mind for a long time.

• 57:35Nathan Labenz?

  Like, almost two years ago now, I did an episode of the podcast with a company called Mosaic LM. And what they were doing, among other things, was this sort of continued pre-training for businesses. They would go into a business and say, let's get all your tokens. You know, and this could be all the Google Docs that you've got and the Slack history and all these various things. Let's compile that and now we can pre-train on that. And hopefully the model will start to speak your, you know, internal native dialect of whatever language you're speaking.

• 58:06Speaker 01

  I call it, as I mentioned, the domain adaptation. Yeah, for example, you have some closed data. Yeah, there's a closed, for example, I have my internal closed data for the documents about my insurance, my customers, about some reports that are not open. And I would like to pre-train the model on this data to make it more adapted to the domain. And we have done such a project now for the biggest bank in Central Eastern Europe, the Pecor Bepa. It's one of the biggest bank in Europe, the biggest one in Central Eastern Europe. And we have performed what we mentioned, the domain adaptation. We adapt the models to their domain. They have said their own data, domain data that are closed data. And we pre-train, continue pre-train the models on their data. And I think it's a very good, very good approach.

• 58:51Speaker 01

  We proved that this, in different tasks, it varies. But there are some tasks that this domain adaptation gives you a huge gain in the quality and in some financial measures and so on. But I think there is one remark I have to mention. I think that only the huge companies has enough data to be able to perform a domain adaptation. Because we know that, for example, you have to at least have, after the duplication and filtering out, You have to reach at least around 10 billion tokens. If you don't have 10 billion tokens, it's not worth to perform the adaptation. And I think if you would like to have the 10 billion tokens in a domain corporate,

• 59:33Speaker 01

  you have at least 30 billion tokens before the duplication and filtering out stage. And the 30 or 40 billion tokens, I think there are only a few companies, maybe not few, but less than 100 in Europe that has 100 billion tokens, they close internal data.

• 59:55Nathan Labenz?

  Yeah, I guess that my intuition is that, I guess it depends what you count, right? I mean, excuse me, I've, I started a company that's, you know, 40 people. And I don't know how many tokens we have, but over all the Slack messages, all the Google Docs, all the Jira tickets, all the, you know, contract proposals that we've sent and the revision history on all of those, I do feel like it adds up pretty quick. So I guess maybe one of the barriers is just like, exactly how deep are these companies willing to mine into their own data? Like if they're actually willing to go get email data from their employees?

• 1:00:35Speaker 01

  I think if you count this data, there may be the plant billions of conversations, the hundreds or thousands of agreements or the proposal agreements. But if you sum up them and they count them and after all, they duplicate them and filter out, maybe it's very hard to get 10 billion tokens.

• 1:00:54Nathan Labenz?

  Yeah, interesting.

• 1:00:54Speaker 01

  Yeah, you can, you can count it. 10, 10 billion tokens is about 10 billion words. And it's, you can imagine that after the duplication, I believe you have at least 30 or 40 billion tokens to have finally 10 billion tokens in a domain corporate. I think it's not so easy. It seems to be easy for many, many, but when we start counting them, it's not so easy to get 10 billion tokens.

• 1:01:18Nathan Labenz?

  And I assume that that sort of what you're counting probably doesn't include like individual employees, email histories, all that sort of stuff. That stuff is like kind of out of scope.

• 1:01:27Speaker 01

  I think the emails, when there are emails between the, for example, the sales forces, for example, the hub, the call center, or the sales emails, you can use them. But for example, there are also some kind of emails that are not keen to be used because of some undefined problems with the intellectual property or the risky cybersecurity risks and so on. I think it's not so easy to use any kind of email. Because always is the reason that some emails are too,

• 1:02:00Nathan Labenz?

  Yeah.

• 1:02:01Speaker 01

  I would say too risky or too, not too delicious.

• 1:02:06Nathan Labenz?

  Sensitive, yeah.

• 1:02:07Speaker 01

  Yeah.

• 1:02:08Nathan Labenz?

  Yeah.

• 1:02:08Speaker 01

  Sensitive.

• 1:02:09Nathan Labenz?

  Yeah, that's really interesting. Yeah, that also does sort of make me wonder if new organization structures are going to be advantaged in some of these dimensions because I totally understand the difficulty that would arise if you said, okay, hey, everybody, and we know you've been working here for all these years and sending all these emails. By the way, we're going to take all that and put it into our training process. You might have a revolt.

• 1:02:37Speaker 01

  The difference that there is, most of them, even the big organizations, they are not aware about what kind of data they have. And before you adapt the AI, or before you train the AI, you have to, first of all, you have to clean your data stores, identify what data you have, what is there creepy or not creepy, the high quality or low quality. And that is the data creation process or data organization process. Everything what is, what is, what is around the topic, how to organize data to find the high quality sub fraction of the data, I think is the problem itself.

• 1:03:16Speaker 01

  Yeah, that the companies, very often the company tried to integrate or deploy AI or even train the AI with all this data accumulation, data organization process. And it usually it collapsed.

• 1:03:32Nathan Labenz?

  Yeah.

• 1:03:33Speaker 01

  I think this is the most important, to be aware about what kind of data inventory you have, what kind of data level of qualities, what kind of data you can use without any regulations, internal regulations, external regulations. And this is the most important step. After this step, when you have the properly identified data sets, good described and well organized, you can go up to invest in the AI training and AI deploying based on that.

• 1:04:06Nathan Labenz?

  That's really all very interesting. Thank you. Definitely great food for thought for me.

• 1:04:12Nathan Labenz?

  What about, just going back, one more question on models. I know you had mentioned that some of the Chinese models have licenses that don't allow you to use them in the EU.

• 1:04:23Speaker 01

  Mostly they were the Mr. Lama 3.4, the first model. In the license, there is a prohibition to use them in the, they are forbidden to use them in the European Union. But now they are KIMI models, the Chinese models, they have the same. They are not able to use them in the European Union. I think it's the problem with the AI Act. Because the AI Act, that was the, the released, the second chapter was released August 2025. They demand from you to create for the general purpose models, the model, the card of model. What data were used on training, how it was secured, what are the data sets used? And what are the resources used for creating them and so many, many other points in the model card?

• 1:05:05Speaker 01

  And I think those, they don't want to evaluate what, what data was used by, by them and what kind of even the telling stages look, look, looks like.

• 1:05:16Nathan Labenz?

  Yeah.

• 1:05:18Nathan Labenz?

  If that weren't a problem, would you be open to using Chinese models? Or are Chinese models not appealing for other reasons?

• 1:05:26Speaker 01

  I think it depends on the task you have.

• 1:05:30Speaker 01

  For example, I will be very, as you say, the sensitive, I will have a huge aversion to the risk when I would like to use the Chinese model to create the long structures of histories or essays and emails and so on. Because I think when you create the longer forms of texts, the ability that the censorship evidence will be, as you say, the easily noticeable is going up. But when you, for example, use some model for the task like the understanding analytical extractive, for example, to extract key information from the documents. Or for example, maybe it's something that you get the information, specific information from the documents.

• 1:06:13Speaker 01

  I am open to use the Chinese model, yeah, because the risk of the censorship in the tasks that are typical analytical extractive ones is very low.

• 1:06:24Nathan Labenz?

  Yeah, that makes sense.

• 1:06:28Nathan Labenz?

  Turning to compute and talent, the other two big legs of the AI stool.

• 1:06:36Nathan Labenz?

  I guess, how AGI-pilled would you say the Polish government is? I mean, it's pretty remarkable that all this is going on at the governmental level already. I would say that that speaks like to a pretty situationally aware and generally agile government. But how, you know, how committed is the government? Or how how big of a deal does the government understand this to be? And, you know, downstream of that are going to be questions around like, how much funding is there? And is there?

• 1:07:06Speaker 01

  There are different, as you mentioned, during the first minutes of our speech, for example, there are three pillars of the AI revolution. The data, as mentioned before, the data organization, data curations. And generally, looking at the data as the crucial point for the AI training.

• 1:07:27Speaker 01

  Next, the compute powers, the GPUs and the AI factories and the others, the DC centers, the data centers and so on, and the talents, it means the people. Yeah, the three pillars, they together combined create the fuel for the AI revolution. And as I mentioned, I think in Poland, we are focused currently mostly on the AI factories. It means to buying as much as we can the GPUs and creating the DC centers for GPU tasks. Of course, we have some projects like the Plume, that is a very good example. One of the, I think, maybe there are three or two projects similar to the Plume in the European Union, that the Ministry of Digital Affairs, they are funds, they have funds, and they funded the consortium of the universities and institutes

• 1:08:12Speaker 01

  that are able to develop the models and the competences and so on. And somehow, they support the talents in this grade we have, because you have the money for the people somehow. But I think we don't be able to be competitive against the US market, yeah? Because in the US market, the AI engineers, they are paid like the NFL players, yeah? I had something like that, the best AI engineers or AI researchers, they have the contracts like the quarterbacks in NFL, yeah? They are at least as the stars. And I think we don't have such a maturity inside the decisions to pay people, to overpay people for their niche competences, because I think it's much harder in European Union, especially in Poland,

• 1:08:56Speaker 01

  to say, how do we, what is the final objective function we would like to get? For example, 10 million customers? Or I don't know, the 10 million dollars each week in our subscriptions? It's much harder for public sector to define such objectives that are very easily monetized and very easily evaluated. And I think in USA, they pay some huge contracts, because they are able to evaluate in somehow, even making some margins, buffers for the future. How this talent can give you what kind of innovation? And how this innovation will pay you back?

• 1:09:39Speaker 01

  And I think this is the problem, that in USA, everything is there. As I mentioned, when I mentioned, I was in Las Vegas, in ETLS. When I mentioned that we are working on the LMS in the public ecosystem, that the Ministry of Digital Affairs, they funded us, we created the consortium, and this is funded by the public sector. All, most of the people I met in Las Vegas, they were surprised that the public sector invest in LMS. But in USA, it's something, it's hard to imagine that the public sector is there. They have enough, I don't know, how to say, the intuition, the enough knowledge, the enough money to invest in such a sexy and very revolutionary topic like AI.

• 1:10:24Nathan Labenz?

  So how do you think this will evolve over the next couple of years? Obviously, the amount of resources that the frontier companies are putting into their current and future models just continues to grow, right? That's expected to be.

• 1:10:38Speaker 01

  And you're kicking door. Even now, we see that. When I'm looking, for example, on the GPT models. And when you compare them, of course, the reasoning abilities are going up in different kind of models. But generally, the GPT 5, it was not such a huge, there's not a huge improvement over GPT 4. Of course, there are some reasoning abilities, but generally, there is a plateau. Yeah, the models are going up, but since some level, they are going, the improvement are very steadily. Yeah, it's like the horizontal improvement, not the vertical one. Yeah, they're just going up, but much more plateau, plateau.

• 1:11:19Speaker 01

  And their development is not so sexy as it used to be. Yeah, remember the ChatGPT 2022, ChatGPT 4, multimodalities 2023, 2024. There are some places in our history of the AI revolution that they were so shocking and they also, they are so making our imagination work so that each year we have something huge, something changing the rules of the game. Now, I think the models are going much more steadily. They are not a huge bumps. But generally, I think now we are starting to counting the costs.

• 1:12:03Speaker 01

  The cost of the energy and the costs of what the models are used for. There is the place of their, I don't know how to say. The same thing is maybe people that when you meet someone or for example, you have some five minutes that you will be adored or maybe not adored. And the same after all, you have to evaluate, is it worth to meet with these people or not? Maybe some five minutes, maybe they were hours. But generally, I think now is coming to the time where we're trying to evaluate what is the real cost of these tools and how we can use them. And if they're used properly, what they give us. Yeah, like the verification stage, yeah.

• 1:12:49Speaker 01

  And I think the verification stage will give us a lot of information that we don't need to have a very huge ARMs. We have to invest in the smart localized ARMs, especially when you are working on previous solutions.

• 1:13:06Nathan Labenz?

  Yeah, I sort of have mixed feelings about that. I mean, on the one hand, I do think already the models that exist are, you know, amazing artifacts for one thing. And very often, especially if you do take the time to do the supervised fine tuning and, you know, really dial in their performance, they can totally work perfectly well for all sorts of use cases. At the same time, of course, we've got the leading companies saying, you know, we're nowhere near done. This is definitely going to keep going. You should expect, you know, more progress. We're going to have AI scientists, we're going to have AI, you know, AI researchers.

• 1:13:45Nathan Labenz?

  How much do you think your strategy depends on or will change if it turns out that there's not so much a plateau, but that you do still see like significant capabilities jumps, albeit with like, you know, even exponentially more resources required to achieve them. Like, how do you, how do you think you navigate that world if it really is the case that? A ten billion dollar training run, you know, is actually that much better than a one billion dollar training run?

• 1:14:16Speaker 01

  I, I think it's much, the most important issue is the first one, the demands and what, what's expected by our customers. Because I, I also mentioned that, that we are very often,

• 1:14:28Speaker 01

  we, we are very often biased by the benchmarks, general purpose benchmarks, but they don't match the benchmarks and the expectation that the business has. And for example, if you have then your business, the company, and you know, for example, you would like to use the AI. And in these places, this AI should have the, at least this kind of metrics. It's really give you the very good information, what kind of benchmarks you should create to evaluate whether, what kind of model is able to reach these benchmarks, this, this expected, this expected metrics. And I think this is the most important, that very often we analyze the general purpose benchmarks, which are, for example, the factual, the reasoning, I know, the, the extractive, the competencies they are evaluating, they are evaluating there.

• 1:15:12Speaker 01

  But for example, for the business, the problem is slightly different. For example, they need the something that write that beautiful email to the customer or something that makes the email that enables you to, to cross selling. That's, that, that very often we don't know what should be done properly because we, there is not business, they don't define the, the requirements very explicitly. I will start from the first and foremost, what kind, what would we like to improve in your business? What kind of tasks you would like to send to the AI? Next, create the benchmarks for these tasks. And next you choose the AI, because I think the most of the business cases I have seen, they don't demand reasoning. You see, you can do it with your, with the, with the normal AI, without the reasoning stages.

• 1:15:58Speaker 01

  I think we should, we should try to, we should, we should slow down a little bit and analyze what is needed to be done and what is especially has the huge business factor. Not only because it's sexy and the public relation likes that, but what gives the money to the business or what makes some savings.

• 1:16:20Nathan Labenz?

  Yeah, there's a massive disconnect, I think, often between general business culture and the culture of things.

• 1:16:28Speaker 01

  There are two trains, but they are not in a sequence, one after one, but they are they are next to each other. One is much faster, the second is going on their pace, but they are not, as you mentioned, very often there is no crossing between them, between them. There are two roads, but the crossing is very far before us.

• 1:16:50Nathan Labenz?

  Yeah. Yeah, that's really interesting. Who are your allies in this? You mentioned, you know, using multiple languages, and I assume that that's sort of in some sort of partnership with maybe other neighboring country national institutes. I guess I'm curious as to how you think the international dynamics will play out. Historically, in the Cold War, we had, you know, the US and the USSR, and, you know, these two great powers were sort of engaged in proxy conflict and whatever all over the world, and a lot of other countries understandably said, you know, this is bullshit from our perspective. And there was a movement of countries that were like, we don't really want to be in either

• 1:17:35Nathan Labenz?

  of your camps. We would rather be just independent and, you know, the beef that you guys have between yourselves, like, we don't really want to be, you know, a pawn in that game. Now it's the US and China, obviously, that are kind of the two, you know, big poles of AI power. How do you think countries that are, I sometimes say, countries three through ninety, three through one ninety-three on the AI power rankings, how do you think they will react? Like, will, do you see alliances forming or, you know, countries working together to share resources, share data sets, to try to create some sort of favored way in the AI space?

• 1:18:12Speaker 01

  There are some movements in the European Union. For example, there are projects that are international, they gather the different type of people from different countries and try to do something together. But I think it's a huge problem that generally, when you would like to get very fast products or very fast outcomes, you have to centralize. The problem is always the same. Generally, the best way is to have the federation. Everything should be spread out, have different people in different countries, they collaborate each other, they build the, how do you say, the wealthiness is going up everywhere, somehow distributed way. But but a normalized way. But generally, when you would like to get very fast incomes, very very fast outcomes,

• 1:18:55Speaker 01

  and and have the products in a month, not in a years, usually you have to centralize the assets in one place. And and there's a problem, yeah, because there are two different ways, yeah. If you you'd like to do it in ideal way, you should create the unions, yeah, the unions of countries, the unions of states, the unions of the partners, the networking, the consortium with the consortium with the plenty of hundreds of stakeholders to get the to get this knowledge everywhere and to be the distribute this knowledge and this power everywhere. But usually, if you have to get some outputs very, very fast, you have to centralize. The problems are, there are two opposite ways, you're not able to do it the same way, both of them.

• 1:19:38Speaker 01

  And I think this is the problem, yeah, because generally, when the, for example, when you have some very huge pressure on the outcomes of new models, you always prefer the centralizations, yeah, that would be Silicon Valley, yeah. You have the huge USA, but most ninety percent of the startups in the Silicon Valley, yeah. This this centralization in one place where there are the money, assets, and every everything. But from the economic point of view, the best place would be distribute these these companies across all the USA.

• 1:20:10Nathan Labenz?

  Yeah.

• 1:20:10Speaker 01

  And I think that that is the problem, yeah, that if you are, if you would like to monetize something, and you would like to get the very fast outputs, you have to centralize. But generally, the best for the economic and and social aspects is to distribute in a normalized way across the the country and across the the continent. What about geopolitical? I think there are still the two players, the Chinese and the USA, they have the two biggest economies. They have money. The Chinese companies, I heard that they pay the same money to the researchers as the U-USA ones, yeah. The the the countries are now currently, they are somehow similar. It means they they pay very well. They don't have to compete, for example, there is not the risk that they will be

• 1:20:53Speaker 01

  taken over by the USA companies because they are well paid in the Chi- in the Chi- China. In Europe, I don't think there is still there, there is a mystery. The European funded startup, now it's not the startup, but it was the startup, two years ago, two years ago. But now I heard that the the the shares have been thirty or forty percent of the shares are they in Microsoft, yeah. There is not so open and as it used to be because there are some stakeholders from the USA. I think the general problem is slightly different. The question is whether the Chinese and USA, they are going to be the rivals still, or there also there is a chance for the cooperation.

• 1:21:38Speaker 01

  This is the question. Maybe it's a chance for cooperation still.

• 1:21:40Nathan Labenz?

  Yeah, from your lips to God's ears. Maybe just one little follow up, and I think this has been excellent. I really appreciate all your time and all these thoughtful answers. Is there anything that you have seen that on sort of a technical or socio-technical level can help with the cooperation of decentralized AI? Here I'm thinking about things like the NEAR protocol. I recently did an episode with the creator of the NEAR protocol, Ilya Plesuhin. Or there's also the intelligent internet, which is Ahmad Mastak's project. There's others as well. These things sort of have this idea that if we create the right scheme, it might be somewhat cryptographically enabled.

• 1:22:25Speaker 01

  There is a topic called federational learning, yeah? Federated learning, that you can use the different data sets somehow anonymized and secured and use them in a way that in networks that you are able to to identify the decisive data, but you can use this data to train your models. Yeah, there are the different kind of ideas about this federated learning. But generally, I don't know that there is some huge deployments of such approaches. Yeah, even though there is a very good approach, yeah, to have such a networks of federations to cooperate each other and to share data in some, as I mentioned, the secure way. But I think still there is, we are not on, if you talk about the business and economy, we are not on the level to use it, yeah?

• 1:23:10Speaker 01

  Because I think we are still on the problems that the companies mostly, they don't know what data they have. What is the quality of their data? What is the value of their data? If you are not able to to measure your own in-house data repositories, how you can go farther and create some data mixture or the network of data repositories?

• 1:23:33Nathan Labenz?

  Yeah.

• 1:23:33Speaker 01

  I think this is that maybe in the future, yeah, in the future, I think there is a chance that there will be the distributed data repositories with the secure levels and optimizations used by the huge consortia to use the as much as we can. But I think this is not the this is not for the next year.

• 1:23:55Nathan Labenz?

  Yeah, so much depends on whether there really is a plateau or whether the frontier companies are just going to continue to keep scaling successfully.

• 1:24:04Speaker 01

  And now I heard that the plateau is caused by because there is a lack of the organic data. Yeah, the biggest companies, they they collected already all almost all organic data that was able to be collected from the internet. Even some of the companies, they they they scan the books that they're not that they haven't appeared in the internet to make this organic data gains relevant. But there is still the problem that I think that there is this repository, the resource of the organic data is almost full.

• 1:24:39Nathan Labenz?

  Yeah, that's why we're now seeing all these simulated worlds and yeah, the strategies to overcome that are going to be definitely fascinating to watch. I for one will bet on them working, but it does, to a certain degree, certainly remain to be seen. Again, really fascinating conversation. It's been awesome to get your perspective. Anything else you want to share or anything we didn't touch on? You want to you want to comment on before we break for today?

• 1:25:05Speaker 01

  I can share that I can recommend to our archive paper, the Plume family models. It is now the archive paper released in November, the first days of November. And I recommend readers and the viewers of this podcast to look inside this paper. The Plume family is the title of this of this paper.

• 1:25:27Nathan Labenz?

  Yeah, that's P L L U M.

• 1:25:29Speaker 01

  P L L, yeah, P. And this, the archive paper, almost 100 pages.

• 1:25:37Nathan Labenz?

  And the P, of course, is for Polish. It's the Polish large.

• 1:25:40Speaker 01

  Polish models.

• 1:25:42Nathan Labenz?

  Marek Kozlowski, this has been amazing. Thank you for being part of the cognitive revolution.

• 1:25:46Speaker 01

  Yeah, thank you very much.

• 1:25:48Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot ing. And thank

• 1:26:33Nathan Labenz?

  you to everyone who listens for being part of the Cognitive Revolution.