Pausing to think about scikit-learn & OpenAI o1

Recently the company stewarding the open source library scikit-learn announced their seed funding. Also, OpenAI released “o1” with new behavior in which it pauses to “think” about complex tasks. Chris and Daniel take some time to do their own thinking about o1 and the contrast to the scikit-learn ecosystem, which has the goal to promote “data science that you own.” Sponsors: Assembly AI https://www.assemblyai.com/?utm_source=practicalai&utm_medium=podcast – Turn voice data into summaries with AssemblyAI’s leading Speech AI models. Built by AI experts, their Speech AI models include accurate speech-to-text for voice data (such as calls, virtual meetings, and podcasts), speaker detection, sentiment analysis, chapter detection, PII redaction, and more. Fly.io https://fly.io/ – The home of Changelog.com — Deploy your apps close to your users — global Anycast load-balancing, zero-configuration private networking, hardware isolation, and instant WireGuard VPN connections. Push-button deployments that scale to thousands of instances. Check out the speedrun https://fly.io/speedrun/ to get started in minutes. Speakeasy https://speakeasy.com/ – Production-ready, enterprise-resilient, best-in-class SDKs crafted in minutes. Speakeasy takes care of the entire SDK workflow to save you significant time, delivering SDKs to your customers in minutes with just a few clicks! Create your first SDK for free! https://speakeasy.com/Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Probabl seed funding announcement https://papers.probabl.ai/announcing-major-milestone-empowering-the-future-of-data-scienceOpenAI o1 announcement https://openai.com/index/introducing-openai-o1-preview/Purdue Data4Good competition https://business.purdue.edu/events/data4good/MLOps Community homepage https://home.mlops.community/ and Slack workspace https://go.mlops.community/slackLatent Space homepage https://latent.space/ and Discord server https://discord.gg/xJJMRaWCRtUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:16Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Learn more at fly.io.

• 0:51Speaker 01

  What's up, friends? I'm here with a new friend of ours over at Assembly AI, founder and CEO Dylan Fox. Assembly AI is where you can turn voice data into insights, chapters, transcripts, summaries, and so much more with their leading speech AI models. So Dylan, give me a glimpse into what you're doing with speech AI models at Assembly AI.

• 1:13Speaker 02

  So at Assembly, we're building industry-leading speech AI models for various tasks like speech-to-text, streaming, speech-to-text, speech-understanding, to help developers easily convert voice data, whether it's live or pre-reported, into super-accurate text. And then to help developers extract a ton of information and metadata around voice data, or even around the text that they just were able to convert from that audio data. So these are things like picking out entities or PII that was spoken in voice files, or summarizing voice and audio data down into custom summaries. It's things like being able to detect how many speakers spoke and who said what and what the names of different speakers were.

• 1:56Speaker 02

  So we bundle all those things into a super simple API with really great docs that developers can just sign up to for free to start, use the API, build into their apps, and then build these really cool AI apps and products and workflows and automations on top of voice data with.

• 2:12Speaker 01

  I dig it. Okay. Can you take me a little deeper into the opportunity for developers? Because it seems like there's a lot of voice data out there and there's a lot of trapped value in that voice data.

• 2:23Speaker 02

  There's so much voice data being created on the internet now, podcasts, videos, phone calls, voice messages, audiobooks, virtual meetings, it's crazy. And you can now transform and understand all this voice and audio data in ways that were not even possible a year, 18 months ago. So what we're seeing with the help of these new AI models that we're creating at Assembly, developers and organizations are just racing to build all these new applications, workflows, automations that leverage the voice data they have either within their organization or with their product to build really cool new products and services, workflows that are just like taking off in the market. So at Assembly, we're building the industry-leading

• 3:06Speaker 02

  models for all those different apps and workflows, whether it's speech to text or speaker diarization or speech understanding capabilities to summarize voice data or extract entities from voice data or mask PII from phone calls for various types of automations that might be built. And we're exposing that through a super simple, super scalable API that's just constantly being updated and constantly getting better. And so we're seeing a crazy amount of developers and companies just build really cool apps and services on top of our API every day. It's really only just getting started, especially with the model updates that we have planned over the second half of the year that are coming out. They're really excited to launch to the developers on our API.

• 3:46Speaker 01

  Okay, constantly updated speech AI models at your fingertips, well, at your API fingertips, that is. A good next step is to go to their playground. You can test out their models for free right there in the browser, or you can get started with a $50 credit at assemblyai.com-practicalai. Again, that's assemblyai.com-practicalai.

• 4:20Daniel Whitenack?

  Welcome to another fully connected episode of the Practical AI Podcast. This is Daniel Whitenack. I am the CEO and founder at Prediction Guard and I'm joined by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. In these fully connected episodes, we try to keep you fully connected with everything that's happening in the machine learning and data science and AI worlds and hopefully share some things with you that'll help you level up your machine learning and AI game. How are you doing, Chris? It'll be fun to catch up on a few things that have been happening over the past couple of weeks today.

• 5:02Chris Benson?

  There are so much going on. Oh my gosh.

• 5:05Daniel Whitenack?

  Always.

• 5:06Chris Benson?

  We'll have to pick and choose what we have time to hit here.

• 5:09Daniel Whitenack?

  How are you hearing about AI things these days, Chris? Maybe that's even something that people might be interested in knowing. Maybe, we've talked about this a few times on the show, but there may be new listeners who kind of got into the show after Gen AI stuff and they're trying to figure out where to keep up with news and learn things.

• 5:32Chris Benson?

  The question's the inverse. It's like, where are you not hearing about it? Because we're getting it from every angle.

• 5:37Daniel Whitenack?

  Or where do you, how do you filter through the noise?

• 5:39Chris Benson?

  That's the issue there. Is there so much noise now? There's so many, like, you know, I, you know, I know both of us have our kind of workflow on how we're consuming new things going on out there and have for many years as we've been doing the show. And I think there's so much more that is coming in and the quality varies hugely in terms of how you might do it. And I mean, I know we're going to talk about a few topics today. And I know that on at least one of those, to be seen in a few minutes, some of the info is kind of straight down the line, gives you the facts and others are very hypey. We've talked a lot about how hypey things are. But I think that's one of the challenges with this field is moving so fast and it's in,

• 6:24Chris Benson?

  it's just one of the dominant topics in mainstream media now. And you have a lot of folks out there reporting on it, some of which know something about it, some of which don't know as much. So yeah, the filtering is the big challenge.

• 6:38Daniel Whitenack?

  It's been slightly different for me, I think, in the sense that Well, one, you know, I founded a company, so my, my attentions are slightly maybe distracted with one thing or another, or maybe exposed to things in different ways. But also, I think my habits online have changed even over the past year or so, whereas I, I was maybe more at least, um, not always posting, but, but looking kind of for things on Twitter or X, um, as it is now, and seeing, you know, a pretty active AI community there, of course, where I had, well,

• 7:21Daniel Whitenack?

  you as well, to, uh, you know, we, we both kind of had different journeys into this, but definitely I had this sort of more background on the data science thing and a lot of kind of data science discussion was at least in my world happening on Twitter or X. Then kind of all the whole AI world took over and also, you know, Twitter changed to X and things are still happening there to one degree or another, but I, I sort of started ignoring that a little bit. And so at least one of the things that we'll talk about today, I kind of looked at and, and learned through LinkedIn, which to be honest, if, if I'm completely honest, I, I

• 8:06Daniel Whitenack?

  always, you know, hated kind of scrolling through LinkedIn, um, throughout the years.

• 8:11Chris Benson?

  Well, it's very corporate-y, yes.

• 8:13Daniel Whitenack?

  But yeah, maybe it's just the, the people that I've learned to follow or look for things from, or yeah, I, I don't know, it's, it's hard to find the proper channel through which you're getting a good signal to noise ratio, but I, I've found a little bit better there recently. I, I don't know if you've seen a similar thing.

• 8:31Chris Benson?

  No, I think you're very right about that, is LinkedIn's gotten better about, uh, producing good information in that way. Uh, and it used to, but once upon a time I had my, my workflow sources and then you'd see it kind of showing up on LinkedIn in the hours and days afterwards. Um, and I also had used Twitter, but I also have moved away from X as well. I just, it's just not giving me what I'm looking for most of the time. And so a bunch of new sources and aggregators put together, uh, with some filtering on that. But yeah, LinkedIn is definitely, uh, on the upswing versus where it was maybe two years ago.

• 9:09Daniel Whitenack?

  Yeah, and, uh, and I don't know, I guess I'm lucky enough to be part of a few Slack channels and Discords now where, you know, either my, my coworkers or, you know, other collaborators or different Discord channels, whether that be, you know, latent space or the ML Ops community.

• 9:32Chris Benson?

  Oh, yes.

• 9:33Daniel Whitenack?

  Like, these things pop up, things as well, and people find them around. And so, yeah, I would definitely encourage people to, of course, to think about, you know, we, we have a Slack channel with the, the podcast, you can find it at practicalai.com/community, but there's also really great ones out there with latent space, ML Ops community, there's, of course, ones, uh, you know, collaborate with your coworkers, figure out where they're finding good info. But, yeah, it is, I think, a little bit more fragmented now, which makes it hard to, to pick apart some of those news stories.

• 10:08Chris Benson?

  I'm gonna say, people need to join our Slack community because, um, we've had some really good conversations.

• 10:14Daniel Whitenack?

  Yeah.

• 10:15Chris Benson?

  Uh, and questions and suggestions arise there. And, uh, the, the folks who are participating really know their stuff. Uh, and they've pointed me, uh, to a few things, uh, in recent months that, uh, have been really useful to have. So, I'm spending more and more time taking pointers from that.

• 10:31Daniel Whitenack?

  Yeah, awesome. Yeah, well, the, one of the first things that I wanted to mention on, on the show here was something that did, you know, I did run across on LinkedIn, which I, I just had missed till it was posted recently as related to their recent seed round of, of funding. But there's, uh, you know, this company, Probable, which had an announcement of, uh, round of seed funding, right? Yeah, I guess they would classify it as seed funding, this sort of, as I've learned, this phases of funding are somewhat strange and all, you know, have different definitions depending on where you're at.

• 11:14Daniel Whitenack?

  But yeah, this is a company probable, which is, I'll use their exact words. Cause I, you know, um, they've, they've chosen them while it's the official operator of the SciKit Learn brand. And, um, they talk about this funding representing a powerful step forward in our mission to help professionals truly adopt our slogan, Own Your Data Science. So a few interesting things there. I am really happy. So we, of course, people are probably saying, well, why aren't you talking about OpenAI? Oh, one as the first thing you're talking about, we'll talk about it later. Don't, don't worry. But this, this one I thought was really interesting in that, you know,

• 11:58Daniel Whitenack?

  hey, I'm, I'm excited to talk about something maybe that, uh, that intersects something other than OpenAI and, and the next gen AI model. But SciKit Learn, of course, has like a huge place in my heart, uh, throughout the years of, of operating in data science. So to see kind of a brand or a company that's really putting an effort behind that brand of SciKit Learn, advancing that and also advancing a slogan forward to the future of, you know, own your data science. One, thinking about kind of owning as in tools that you can operate internally and privately and with your own data,

• 12:42Daniel Whitenack?

  but also tools that are around data science. So, you know, maybe there's a future for data science after all.

• 12:49Chris Benson?

  I think there is and, uh, and I love, you know, the reason we put this first is because as huge supporters of open source and people being able to kind of control their own destiny, uh, with data science and machine learning and AI technologies. I love having these companies that are out there supporting open source and it gives us, uh, options. And so, you know, if you just want to do the open source yourself and maybe you're just a data scientist, uh, working on your own side project or something, you can do it. Or if you're a corporation and you're looking for, to have, you know, dependable partners to work with around open source.

• 13:29Chris Benson?

  You've got that too. So it makes it, in its own funny way, it makes it more accessible to a wider audience and gives us those choices. And I just love it when companies are doing that. You know, just as a two second aside, it's one of the things that surprised me, you know, with Facebook and their models is they're open sourcing it, you know, which the other big companies we've historically talked about aren't. But back to probable, what are some of the things you see in this, in this announcement that really gets you going, Daniel?

• 14:01Daniel Whitenack?

  Yeah, well, I think, you know, for people that, that aren't aware, and maybe just coming into kind of the AI world with, you know, ChatGPT and all of that, scikit-learn has been around for, for a long time and is kind of a primary open source set of tooling for the, the Python community and the data science community that allows you to build a wide variety of models. So everything from neural networks to decision trees to random forest models to support vector machines and clustering algorithms, just a huge number of sort of algorithms and metrics

• 14:42Daniel Whitenack?

  and evaluations and models, all within this very kind of consistent API, well-supported API, widely used library that is scikit-learn. And so I think I'm excited to see that, you know, one could think, well, maybe everyone that was interested in supporting and contributing to those things has kind of jumped ship to what's, you know, fancy and shiny with LLMs and all this stuff. But I think I'm encouraged that there is a strong backing behind this, and I think it's needed moving forward. We've talked about this on the show where- There is going to be a need for kind of hybrid systems between traditional machine

• 15:27Daniel Whitenack?

  learning and statistical learning and generative AI. And there will still be a need for kind of smaller performant models in a variety of contexts. And maybe those are better, faster, more secure for a whole variety of things as they've been useful for, you know, decades now. So I think that I'm encouraged for what's there. Just so people know, and we'll link to this announcement, but they they talk about what's next for for probable. So one of the things they talk about is an aqua hire, securing talent, but also the

• 16:07Daniel Whitenack?

  launch of an official psych at learn certification program coming in Q four 2024. And then the release of a product. So obviously, this is a commercial company, so they will have something productized. And they say that they're aiming at augmenting the work of data scientists in the pre ML Ops phase. So this is very interesting to me. I think it's an interesting niche that they're that they're focusing on there, not trying to cover or, you know, recreate ML Ops tooling that's already out there, but, you know, focus on the data scientist role in that kind of pre ML Ops period, which in my mind,

• 16:48Daniel Whitenack?

  you know, involves, of course, data munging and model selection, feature development, all of these sorts of things.

• 16:57Chris Benson?

  You know, I think that, and it's funny that you bring that up because I think that gets lost, you know, with with all the hype on the AI and particularly Gen AI these days, you know, there's still so much data science going on. And I would I would suggest that just core every data data science is still by far bigger in terms of being present in the number of organizations. It may not be a big fancy glitzy effort, but it's pretty core to most organizations. And yet the AI hype tends to get all the press. So it's really, you know, it's really good to see them kind of acknowledging that that that section is still there and that it does need support and being willing to

• 17:41Chris Benson?

  do that a little boldly and stepping a little bit away from where everybody else is going to. So yeah, I hope they do really well in that capacity.

• 18:02Speaker 01

  Okay, friends, I'm here in the breaks with Annie Sexton over at Fly. Annie, you know, we use Fly here at ChangeLog. We love Fly. It is such an awesome platform and we love building on it. But for those who don't know much about Fly, what's special about building on Fly?

• 18:18Speaker 05

  Fly gives you a lot of flexibility, like a lot of flexibility on multiple fronts. And on top of that, you get, so I've talked a lot about the networking and that's obviously one thing, but there's various data stores that we partner with that are really easy to use. Actually, one of my favorite partners is Tigress. I can't say enough good things about them when it comes to object storage. I've, I never in my life thought I would have so many opinions about object storage, but I do now. Tigress is a partner of Fly and it's S3 compatible object storage that basically seems like it's a CDN, but is not. It's basically object storage that's globally distributed without needing to actually set up a CDN at all. It's like automatically distributed around the world. And it's also

• 19:03Speaker 05

  incredibly easy to use and set up. Like creating a bucket is literally one command. So it's partners like that, that I think are this sort of extra icing on top of Fly that really makes it sort of the platform that has everything that you need.

• 19:17Speaker 01

  So we use Tigress here at ChangeLog. Are they built on top of Fly? Is this one of those examples of being able to build on Fly?

• 19:25Speaker 05

  Yeah, so Tigress is built on top of Fly's infrastructure, and that's what allows it to be globally distributed. I do have a video on this, but basically the way it works is whenever, like, let's say, a user uploads an asset to a particular bucket. Well, that gets uploaded directly to the region closest to the user. Whereas with a CDN, there's sort of like a centralized place where assets need to get copied to, and then eventually they get sort of trickled out to all of the different global locations. Whereas with Tigress, the moment you upload something, it's available in that region instantly. And then it's eventually cached in all the other regions as well as it's requested. In fact, with Tigress, you don't even have to select which regions things are stored in. You just get these regions for free. And then on top of that, it is so much easier to work with.

• 20:09Speaker 05

  I feel like the way they manage permissions, the way they handle bucket creation, making things public or private, is just so much simpler than other solutions. And the good news is that you don't actually need to change your code if you're already using S3. It's S3 compatible, so like whatever SDK you're using is probably just fine. And all you got to do is update the credentials. So it's super easy.

• 20:32Speaker 01

  Very cool. Thanks, Annie. So Fly has everything you need. Over 3 million applications, including ours, here at Changelogue, multiple applications have launched on Fly. Boosted by global Anycast low balancing, zero configuration private networking, hardware isolation, instant WireGuard VPN connections, push-button deployments that scale to thousands of instances. It's all there for you right now. Deploy your app in five minutes. Go to fly.io. Again, fly.io.

• 21:29Daniel Whitenack?

  Well, Chris, one of the things, you know, we started out talking about probable and this funding that they have, of course, this is related to scikit-learn and owning the data science process with scikit-learn, they're obviously very committed to the open source way of going about things. They're, you know, have a long-term vision for people still to be able to, to maintain their own data science process with their own data, with models that they own internally. That's of course in, in contrast to some things out there, which, you know, there, there's a mix of this. I think there is a validity to

• 22:13Daniel Whitenack?

  people trying to create their own proprietary models and that's their way of owning certain things. But certainly the models that people would think of right now with AI models or maybe those from OpenAI. And, you know, we got another one of those over the past, whenever it was, I don't know, all the days blurred together for me. But this rumored, what was it? Sort of strawberry code name slash O1?

• 22:44Chris Benson?

  Correct.

• 22:45Daniel Whitenack?

  Model, which people have been talking about. And certainly we want to, we want to cover that on this news show. So yeah, O1, Chris, how has O1 struck you in its first days on the AI scene?

• 23:01Chris Benson?

  Yep. It's an interesting animal. And, and, you know, it is entirely proprietary. And, and on this show, as we've noted, we usually give those the, the second spot, not the first spot. But it's interesting. I've been, I've been using it some over the past few days. It's, you know, some of the new features that it talks about are advanced logical reasoning, and it has a capability where it slows things down.

• 23:30Chris Benson?

  In terms of of processing, and so that that can be, it's not the same experience as the 4.0 that we've been used to where we've been, you know, I know on my iPhone, I on 4.0, when I'm using that, I'm talking at it these days and it talks back and it's fairly conversational due to the speed. Can't really do that with this 0.1 preview as it's currently released. It's a little bit too slow for that. When I've tried, I've had to wait a while for my responses, but it's taking a different approach.

• 24:01Daniel Whitenack?

  Yeah, I guess it brings up an interesting question. So for people that are not familiar with the 0.1 model, it's a model that really operates on this principle of thinking through a series of stages of reasoning before giving a kind of final answer. Some of what this might have been called in terms of how a completion would happen, or how you would structure a prompt or a completion in the past might have been chain of thought processing. And so that takes a bit more generation, more text is generated, you know, there's a pause in the result. And this brings up a question, you know, you were talking about this latency

• 24:44Daniel Whitenack?

  element, Chris, and here they've intentionally slowed things down. And I was wondering for a while, you know, with Grok and very quick, you know, completions out of these models, does what role latency would really have, you know, at a certain point, as text is generated, our human minds can still only process, you know, a certain amount of natural text at a certain speed. And then here, you know, OpenAI has intentionally slowed down the generation process, which I don't know if you'd call it, maybe it's still generating at the same or a similar token per second rate. But it's generating more, right? It's generating this kind of chain of reasoning or chain of thought kind of steps,

• 25:28Daniel Whitenack?

  and then producing something on the output, which I'm assuming is just like a generation in, and there's then some special token that they have in their prompts that they train the model on, which is like, now give the answer to the user, right? Whatever that special token is, which, you know, we don't have a ton of information about. But yeah, I don't know. What are your thoughts on this kind of latency versus reasoning versus also the human interaction element of this?

• 25:58Chris Benson?

  Before I dive into that, I want to, I want to note that having read quite a bit about the model over the last week or so, a lot is a bit speculative in terms of, you know, you'll read articles where people are saying, clearly it's doing X.

• 26:12Daniel Whitenack?

  Yeah.

• 26:12Chris Benson?

  But they don't really know because OpenAI has not specified, you know, exactly how they're doing it. So I want to note.

• 26:18Daniel Whitenack?

  Yeah, so any, anything we say here may or may.

• 26:21Chris Benson?

  May or may not be accurate.

• 26:23Daniel Whitenack?

  These are our best, this is our best chain of thought on the topic.

• 26:27Chris Benson?

  And I, I think it's important to say that we're not speaking with, you know, in perfect factual.

• 26:33Daniel Whitenack?

  We have no direct line to OpenAI's technical team and revealing their secrets.

• 26:38Chris Benson?

  Nor do many of the authors that we've been talking about. So I, I guess my impression has been, you know, it's a little bit startling. You know, the latency thing kind of kicks in after you've been using the, the 4-0 model for a while. And it's, and, and it forces you to start realizing that there are definitely different use cases for using the 4-0 and the 4-0-1 preview. And I think that that is notable because it's really the first time that OpenAI has offered a new model that everyone didn't just instantly switch to that as the thing to use. It was kind of like, they said when, when we went from 4 to 4-0, well, there may be cases where you go to either one.

• 27:20Chris Benson?

  But in practice, I saw people just going to 4-0 pretty much non-stop at that point. Whereas this one, the latency kind of forces you to change your ways. And they also have given. Guidance on the prompt engineering that the way you prompt the O1 preview is not the same anymore because what they're doing behind the scenes and you know with the the notion of possibly multiple concurrent addressing of your prompt in different ways and they verify them against each other. All things that I've read that are unverified at this point in time. That because they've changed the the way they're processing on the back and the latency is is now there that there might be different types of things. They tend to highlight coding.

• 28:05Chris Benson?

  They tend to highlight math and other critical reasoning skills where you'd want to go to the O1 preview rather than back to the 4O. And they're claiming that it's more accurate. I've seen numbers like 15% increase in complex reasoning tasks to support that. And there's some drawbacks which we can talk about in a moment, but I think I am still trying to figure out the tasks that I'm going to assign to each model in my own mind and I'll find myself stumbling a little bit on that. I probably at this point over the next week, trying this stuff out when I'm coding. I don't have a reason to do a lot of math in my day to day stuff just on a on a constant basis.

• 28:47Chris Benson?

  But for coding, I'll probably be spending more time on the O1 preview than I whereas I was using the the O itself before that. So it'll be interesting to see. And so with that difference, as you look back to the two between the O1 preview and just the 4O we've been using. I just want to throw out into the mix. At some point in the not too distant future, we've been led to believe that GPT-5, which is a much larger model, according to OpenAI, will be coming out. And so it kind of leaves you wondering a little bit, you know, is that which model that we're on now? Is that going to be closer to? Are there two different sets of kind of prompt engineering approaches that we have going forward at this point?

• 29:32Chris Benson?

  There's a lot of unanswered questions.

• 29:33Daniel Whitenack?

  Yeah, and it was, you know, kind of curious to me and maybe slightly revealing. I don't know. It's hard to read into everything that's going on behind the OpenAI curtain. But the fact that the sort of big advance here was apparently some sort of, you know, RLHF preference tuning around this sort of chain of thought generation versus kind of a diff, you know, you could think of any number of things that that could change, you know? We saw this in the past, we saw a wave of mixture of experts' models, right?

• 30:14Chris Benson?

  Yes.

• 30:15Daniel Whitenack?

  You know, that was, that was a big change in the model. We saw, you know, of course, the model size. We saw, you know, changes in, at a certain point in terms of how models were trained and aligned. But here, this is just sort of a different prompt set, you know, that is used in this RLHF process. And I think it's intriguing and interesting that they're applying it a different way in the UI and the, you know, ChatGPT UI.

• 30:46Chris Benson?

  You might want to define the acronym as you're going there, just.

• 30:49Daniel Whitenack?

  Yeah, so like, this is all inference on my own, with my own chain of reasoning. But when they say sort of, oh, we're quote pausing the model in the generation, like, we're allowing it to think and then all, I think that's somewhat confusing because obviously the model is not, you know, well.

• 31:10Chris Benson?

  There's a little marketing thrown in there.

• 31:12Daniel Whitenack?

  Again, this is my own personal opinion, but the, the model is not thinking. It's just generating text, right?

• 31:17Chris Benson?

  Yeah.

• 31:18Daniel Whitenack?

  And it's generate, the difference is that it's generating text that is more explanatory or exploratory, representing a series of decisions that can be made, leading to accomplish the goal or the task that you prompted it to accomplish. So when it's pausing like that, what I'm assuming is that they have a pre-trained model. Maybe it's the GPT for something internally. I don't know what they call it internally, the parent model, right? And then they've curated a set of prompts that in the training set, complete with the complete,

• 32:00Daniel Whitenack?

  you know, chain of thought, whatever, whether they synthesize that data or used humans to create it, or, you know, whatever, probably some combination of that, right? And then they fine tune or preference tune or align the pre-trained model to that prompt set that they've curated, which includes all of this chain of thought stuff. Using this process, I mean, they refer to RLHF, reinforcement learning from human feedback. So this sort of reinforcement learning-driven loop to align the model. And then you get the sort of O1 model. That is the process I am assuming happened on the backend.

• 32:41Daniel Whitenack?

  And when you're in ChatGPT, you know, it's not like there's some button, like, you know, they have it thinking, thinking. They say that. Really, it's just, I'm assuming it's generating text. And then at a certain point, similar to like, people ask like, these models just generate text. How do they know when to stop? Well, they don't know. They generate a special token, which is an end of statement token. And then the program, the computer program stops it from generating more text after that token is is generated. In the same way here, I'm assuming after a certain level of text is generated in the chain of thought, it generates a special token. That's what I was referring to before of like, now show the answer.

• 33:25Daniel Whitenack?

  So, you know, now generate the answer token. And then that's how they control the UI. So all of that is inference on my part. Again, like, I could totally be wrong, but that really is a similar process to what we've been doing now for a couple of years with these models. There's not anything fundamentally different about that process. And so I I found it interesting that the big reveal here was sort of. A different model created with RLHF, which is basically what everybody else is doing.

• 33:59Chris Benson?

  Right.

• 34:00Daniel Whitenack?

  Maybe they're using this like interesting methodology and the UI. So I don't know what that reveals. It could mean like this is a cool thing to hold us over until GPT five, which will be this fundamentally world changing, different process methodology, architecture, model. That's going to be crazy. Or it could just mean like there's a very much a diminishing returns here in terms of the methodologies that are available to improve this wave of models.

• 34:31Chris Benson?

  I, I think that that's as, as good of an educated guess. You know, not privy to their, to their internals as, as I've heard. So I, I think if you haven't nailed it completely, you've probably nailed parts of it. This model being out there in the consumer market. So on my iPhone, I use it a fair amount. I find it interesting that they're trying to create a user experience that's a little bit, a little bit mystic, you know? As you pointed out, it says thinking while, while the delay is going on, which kind of, there's an implication there, especially if you're not, if you're not like us and in the industry where we're talking about AI every day and that's what we do. If you're somebody out there who's just a kind of just your typical average person

• 35:14Chris Benson?

  consuming the technology, there's an implication there. And especially when you talk about they're using the word advanced logical reasoning, things like that, that I think there's, it's a little bit of, of kind of marketing hocus pocus that's being applied to, to a fairly mundane set of processes. As you, as you pointed out, using the technologies that have been, they may be, you know, constructing how their models are interacting, as you pointed out, in a way that's unique on their side, but it's probably not revolutionary. It's an evolutionary decision that they've done to try to make their model more accurate. I find that a little bit questionable in terms of kind of how they're marketing that to the general public.

• 35:56Chris Benson?

  A little bit worrisome, there's some of the, the voices out there that have been out, you know, talking about this coming out or expressing concern. Again, the, the lack of visibility into what's actually happening makes it really hard to verify or not, you know, how they're approaching.

• 36:33Speaker 01

  Well, our friends over at Speakeasy have the complete platform for API developer experience. They can generate SDKs, Terraform providers, API testing docs, and more. And they just released a new version of their Python SDK generation that's optimized for anyone building an AI API. Every Python SDK comes with PyDantic models for requests and response objects, and HTTP/AX client for async and synchronous method calls, and support for server sent events as well. Speakeasy is everything you need to give your Python users an amazing experience integrating with your API. Learn more at

• 37:14Speaker 01

  speakeasy.com/python. Again, speakeasy.com/python.

• 37:29Daniel Whitenack?

  Well, Chris, just to give people a sense of this O1 thing, and then I think could get maybe a last impression from your end, and then maybe a summary. But I was trying one. So I think the idea, at least, or part of the idea with the way that they're providing access to this model and promoting its usage is for more kind of researchy, kind of deep reasoning type of things. So the simple prompt that I gave was, "Determine a new problem in physics related to density functional theory for my PhD focus. Write a concise

• 38:13Daniel Whitenack?

  summary for me." So that was my prompt. The only reason I did that prompt is because that was the subject of my PhD research. I'm like-

• 38:22Speaker 01

  You know something about it there.

• 38:23Daniel Whitenack?

  Hey, you know, this would have maybe been nice back in the, back in the day. And so the UI experience, for those that have not tried it yet, and you're just listening- It kind of paused, said thinking for about 10 seconds. Then it gave the executive summary or the summary that I asked for, but there's a little kind of dropdown that I can click and it said thought for 10 seconds. And the steps that it says that it kind of talked through or formulating a research problem, breaking down DFT applications, considering quantum embeddings, changing gears and pioneering new DFT modeling.

• 39:08Daniel Whitenack?

  And it gives a little summary of those and seems to be, you know, more text generated there and that sort of thing. So the problem statement is relevant. The text is relevant to like some of the things that I would know about in that case, but also pretty much what I'm aware of from what people have already been exploring in the past. It's not like a, all of a sudden this model knew a, like it shocked me with a wow, that would be a really interesting and profitable area of research in this topic that I'm aware of. But you know, it was generally in an area that's interesting.

• 39:49Daniel Whitenack?

  So I guess, you know, kind of good-ish, but not mind blowing. And so, yeah, that, that kind of brings up the part of what I'm wondering here around where is the proper place for this in my day-to-day workflow? Maybe we just haven't figured that out yet because I can definitely see a lot of places where I'd be like, well, I don't want to pay up for this because it is, you know, going to be way more expensive than like the 4O or 4O mini, right? I could see a lot of places I could apply for 4O mini or any number of LLMs that would operate in a similar way. Where am I going to apply this in kind of, in the software that I'm building? I don't know. I haven't really pinpointed that yet.

• 40:34Chris Benson?

  You know, and I agree with you. I want to go back to something you mentioned though, as you were talking your way through the example. And that's that it gives you these kind of intermediary steps that it says it's following along the way. And from my standpoint, I think that's more, once again, on the marketing side, tried to reinforce that, that reasoning, marketing message that they're, that they're driving on that. You know, as we talk about where things, what's an appropriate set of tasks to use with each of these models, and especially with this O1 preview that's out right now, it warrants knowing that there are a few drawbacks to the model, which we have not called out yet. One of them is that it still has a cutoff date on knowledge,

• 41:19Chris Benson?

  which is tied back to October of 2023. So, you know, it's almost a year since it has access to that knowledge. And unlike some of the other models before it, which had internet access to go out and kind of make up for that and get some more current information to throw into the model that was trained up until the cutoff date, this one does not have the ability to browse the internet. So that alone may change kind of how you use it. Because, so if you were, you know, going back to what I was suggesting, if I'm coding, well, it may be that if I'm worrying about whether it's Python code or Rust code or Go code, not a huge amount has changed in the coding world in that time

• 42:02Chris Benson?

  in terms of what's available library-wise, unless it's just the latest, greatest thing to come out. And so I can probably use it really beneficially in a coding context there. But as we record this, for instance, on this model, just to talk about current events, a potential second, it sounds like a second assassination attempt on Donald Trump occurred today in the news. And if I wanted to ask a model to get information about that, this would not be the model. And I'm just pulling that one out simply because it's a big news event that happened on the day that we're recording. And so in that case, if I'm curious to explore the story or whatever, using a set of models, I might have to go to some of the other models that have the internet access

• 42:45Chris Benson?

  and might be able to frame things that I'd be curious about for my consumption of my knowledge, whereas that current event would not be applicable here. And then finally, I wanted to point out that we're kind of used to being able to upload into the 4.0 series, you know, the external documents from a RAG perspective, retrieval augmented generation. That's not available in the preview here. So that's yet another limitation. And when you add up the cutoff date, you add the lack of file upload, and you add the lack of internet access. Those are some fairly substantial limitations on this model at the current time. So there truly are certainly a set of use cases that you might go to different models

• 43:30Chris Benson?

  to see, you know, current events. It's not going to be this one.

• 43:33Daniel Whitenack?

  Yeah. To kind of wrap up this section here, Chris, on 01, bringing it back to scikit-learn and the probable funding. I found it interesting that probable on their website has a bit of a manifesto as far as their values. And maybe I can share these and, you know, I'll leave it to you and the audience to see how these compare in this sort of scikit-learn ecosystem, open data science, own your own data science, and the open AI world of AI. So their values, they

• 44:14Daniel Whitenack?

  talk about supporting the whole long-term ecosystem than individual stakeholder gain. Openness rather than proprietary lock-in. I think that's definitely an interesting one, especially, you know, I was talking to someone the other day even about this idea of model lock-in in the AI world in this proprietary sense. The other ones are interoperability rather than fragmentation, cross-platform rather than platform-specific.

• 44:47Daniel Whitenack?

  Collaboration rather than competition, accessibility rather than elitism, and transparency rather than stealth. So if you're interested in any of those types of values, definitely check out the data science community around scikit-learn and other projects and give a shout out to what they're doing, what they're doing over there. So I think it's an important, important balance and cool effort to highlight in, in light of all the, the other crazy things happening in our AI world. So.

• 45:26Chris Benson?

  I think those are fantastic values just in general. And they're very consistent with other open source and kind of commercial support of open source that we've seen in other companies that we've liked, whether they're in the AI space or the software development space. And I find myself certainly feeling very comfortable and gravitating, but I would actually conclude by saying, in my own experience, working at some large companies, you know, large corporations and, and having a lot of friends and colleagues that I talked to at other big companies, they may use proprietary models to some degree, but nobody's betting their business. And at least in the conversations I have, they're not betting their business

• 46:09Chris Benson?

  on an entirely opaque proprietary approach. So you see it there in use cases, but not for the big things. For the big things, they're looking at, at open source models that they can rely on and such. And I just wanted to call that out because that's been really notable to me over the last year or so, is how strong that sentiment is. So, and I would say more power to it.

• 46:35Daniel Whitenack?

  Yeah, and we, we do like sharing some learning resources and experiences here on the show.

• 46:42Daniel Whitenack?

  And in particular, as related to scikit-learn and in that community, if you're interested in those things, you can take a look. There's some learning resources around DataCamp. So DataCamp has a supervised learning with scikit-learn course, which I think you can try out for free. I think Codecademy as well has a course there. There's probably innumerable blog posts around the internet in terms of scikit-learn and what you can do with it. So maybe if you're coming from the gen AI world and have tried a bunch of things with with OpenAI, you could also, you know, dip your toes a little bit into the data science world.

• 47:26Daniel Whitenack?

  We're happy to welcome you in and try a few things with scikit-learn. They, of course, have great documentation and examples on their on their site as well. I'll also mention, if you're hearing this podcast right after it airs, I think there's still time for you, but Purdue University, which I'm I'm close here, but they're working with actually a a bunch of partners, including commercial partners, but they're running a Data for Good competition, which is close to Chris and and I's heart. So they're they're supporting the TAPS organization, which provides grief, counseling and care for those that have lost family members that served in the military.

• 48:11Daniel Whitenack?

  And so if you're interested in that, check it out. That's for graduate students and undergraduate students. And there's a huge, I think, $45,000 in cash prizes, but also a bunch of free training and AI that you'll get along with the the effort. So I really encourage you, if you're a student, that's a great thing that's happening this fall that you could be a part of. So check it out. If you just search for Data for Good Purdue, the the link will be there. And I think you could register, you know, if you hear this right after it airs, go check it out and and sign up right away.

• 48:47Chris Benson?

  Fantastic.

• 48:48Daniel Whitenack?

  Awesome, Chris. Well, good to talk things through and enjoy the rest of your week until we hear about O2 or GPT-5. We'll we'll see you next time.

• 48:59Chris Benson?

  There's always another one coming. See you next time.

• 49:05Speaker 00

  Thank you for listening to Practical AI. You know what's cool? Free stickers. During the month of September, we're mailing out Change Log sticker packs to everyone who leaves us a thoughtful five-star review or blog post about our pods. Simply email proof of your review to [email protected] alongside your address, and we'll mail out the goods anywhere in the world. Once again, that's [email protected], picks or didn't happen only in the month of September. Let's do this. Thanks again to our partners at fly.io, to our beat freak in residence, the one and only Brickmaster Cylinder, and to our longtime sponsors at Sentry.

• 49:49Speaker 00

  Use code CHANGELOG when signing up for a new Sentry team plan and save $100. That's all for now. We'll talk to you again next time.