Goldman CIO Marco Argenti on the Warp-Speed Improvements in AI

When we last spoke to Marco Argenti, chief information officer at Goldman Sachs, we were talking about how the bank was deploying AI, including the development of its own internal tools. But that was a year and a half ago and a lot has changed since then, especially with the arrival of agentic platforms like Claude Code. So what exactly is Goldman Sachs doing with AI now? And what has its experience with the new tech been like so far? On this episode, we catch up with Marco to discuss what AI deployment at the bank actually looks like at the moment — including how AI coding is changing the work of its developers and engineers — to all the data challenges and regulatory concerns that come with integrating this technology at scale. Subscribe to the Odd Lots NewsletterJoin the conversation: discord.gg/oddlotsSee omnystudio.com/listener for privacy information.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Joe Weisenthal?

  There are some market stories where you want every detail. Tracy and I have made quite a few podcasts on that basis, but sometimes you've only got 10 minutes and need to know what's moving markets and why. That's the Barclays Brief Podcast. Every week, experts from Barclays Markets and research get you up to speed on what matters and what to watch next. About the time it takes to grab a coffee, so search Barclays Brief wherever you get your podcasts.

• 0:26Speaker 01

  This is the Bloomberg Tech Minute brought to you by ChatGPT. I'm Carol Muster. Getting hired at SpaceX requires exceptional ability and may include having Elon Musk review a candidate's resume. So writes Bloomberg Business Week's Lauren Grush. She reports that the company culture is defined by Musk and job candidates must understand that a commitment to SpaceX means adhering to this culture and exhibiting loyalty to Elon Musk. To get hired, candidates must demonstrate their achievements from past jobs and hands-on experience matters more to recruiters than advanced degrees, with the company looking for people with raw, innate intelligence. A spokesperson for SpaceX didn't respond to a request for comment on its recruiting

• 1:07Speaker 01

  and hiring practices, but according to recruiters, advisors, and those who've interviewed for jobs there, it's the space exploration dream that draws people to the company, and Musk invokes that dream regularly to remind workers why they're there. Prospective hires are told that the job will require stamina, including in job ads that candidates, quote, "must be willing to work extended hours and weekends as needed." One major perk for all that hard work? Employees' access to equity, either the right to buy shares or payment in the form of restricted stock units that vest over time. That's the Bloomberg Tech Minute brought to you by ChatGPT. Get started at chatgpt.com.

• 1:51Speaker 01

  Bloomberg Audio Studios. Podcasts, radio, news.

• 2:06Tracy Alloway?

  Hello and welcome to another episode of the Odd Thoughts Podcast. I'm Tracy Alloway.

• 2:10Joe Weisenthal?

  And I'm Joe Weisenthal.

• 2:11Tracy Alloway?

  Joe, the thing about AI, I feel like it's accelerated all of our timelines, right? Like, it's phenomenal to me to think back that, like, back to the days of ChatGPT. And when did that come out? 2020?

• 2:25Joe Weisenthal?

  Late 2022.

• 2:26Tracy Alloway?

  2022?

• 2:27Joe Weisenthal?

  November 2022.

• 2:28Tracy Alloway?

  That's just crazy to think.

• 2:30Joe Weisenthal?

  It's really unbelievable, the gap. And I've been thinking about this, like, just, A, the explosion of capabilities.

• 2:37Tracy Alloway?

  Yeah.

• 2:37Joe Weisenthal?

  And the thing I've been thinking about is that, you know, after the first year or so when it came out, we talked to executives. And we're like, how are you using AI in your workflow? And they're like, well, everyone's experimenting. It's great. Everyone is using ChatGPT. It's always very vague. And now in 2026, the story is that AI is so powerful that it's going to destroy all these legacy software companies. So what I would say is, we must be past the age of experimentation. I think that any company

• 3:02Tracy Alloway?

  To real use cases, yeah.

• 3:03Joe Weisenthal?

  Using it better have some example of like, here is a workflow where we're using it.

• 3:08Tracy Alloway?

  Well, exactly. And to this point, now that we're past the age of experimentation, I'm very curious how executives and managers are actually evaluating the return on investment in AI and what they actually want to see from it at this point. So, you know, are you going to replace all your third-party SaaS contractors with internal coders? And what does that look like from an actual headcount perspective, from a cost savings perspective? We can actually get some concrete details on this now. So I'm very excited to say we do in fact have the perfect guest. Someone who we had on before to talk generally about AI and someone at a company that has been doing, you know, they got into it pretty fast.

• 3:50Tracy Alloway?

  The last time we spoke to this person was in 2024. And even since then,

• 3:54Joe Weisenthal?

  Ages ago.

• 3:55Tracy Alloway?

  I guess. It just feels like light years in AI time.

• 3:59Joe Weisenthal?

  So just one last thing on the last few years of AI, which is that when ChatGPT I used, when it came out, I played around with it a lot when I had to write poems and all this stuff. And then I bet if you Actually looked at my AI usage, went for through a trough. Whereas like I wasn't really getting any productivity. There was nothing it really could do that I needed. It still sort of seemed like a toy. So I had this like intense burst of use for the first several months. And then this trough. And now these days with the expansion of capabilities, particularly Claude Coda, finding all kinds of new things. So there is like, we're out coming out of the trough. I think a lot of people are actually finding things, at least if I can generalize from my own experience.

• 4:37Tracy Alloway?

  Yeah, absolutely. So we do in fact have the perfect guest. We've brought back Marco Argenti. He is of course the chief information officer over at Goldman Sachs, someone we had on the podcast back in August of 2024. So Marco, thank you so much for coming back on.

• 4:53Speaker 03

  Thank you for having me.

• 4:54Tracy Alloway?

  How much have things changed for you? Does 2024 seem like 20 years ago now in AI time?

• 5:01Speaker 03

  Yeah, I barely remember even what what happened back there.

• 5:05Tracy Alloway?

  That's a nice way of saying you forgot what we talked about on the podcast. That's fine.

• 5:08Speaker 03

  Maybe that. But literally like things are really changing on a weekly basis almost right now. And if I look at the evolution, not only since a year ago, but even since six months ago. I think it has been nothing short than revolutionary. A year ago, we barely talked about agents or the word almost didn't exist. We were using AIs like a chat companion.

• 5:31Tracy Alloway?

  Yeah.

• 5:32Speaker 03

  There was.

• 5:32Tracy Alloway?

  A search function.

• 5:33Speaker 03

  Yeah, it was telling you, oh, I'm sorry. I don't know who's the president of the United States because my cutoff date is like a year and a half before or things of that nature. Now you can say, hey, as a person, you can say, hey, my, my plan just got canceled and it's going to redo all your plans. It's going to check for like available flights. It's going to do all these capabilities of personal assistant. And that translates in corporations also in a lot of utility that you can see in everyday's tasks. So I would say, you know, what I like to say to my people about us in general, I say, this is not the drill. This is real, you know, it's not the age of experimentation anymore. This is a tool that now can do a lot for you.

• 6:13Speaker 03

  And so we put it to work and we put it to work, starting from developers, but then expanding in many, many other areas. So I would say. Actually, if I look at the increase of capabilities of these models, what we've seen in the last six months or so with really the evolution of this advanced reasoning capabilities they came out. I think that finally got us the confidence that you can use AIs for every day's work with the right supervision and also in many cases for mission critical applications. It's not a toy anymore. It's something that you can expect results from, and I think that's the biggest change. So today, I would say that there is almost nobody in Goldman that is not touched in a way or another by AIs.

• 6:54Speaker 03

  We gave our GS AI or GSA assistant to forty seven thousand people. Most of them use it every day. Most of them use multiple times a day. And what's interesting is it's like the first time you see a tool like, I don't know, Microsoft Excel. You can almost not predict what people are going to do with that. Okay, maybe it's born for doing some form of accounting and then people write entire applications on top of that or use it for project managers, management or things of that nature. And AI is kind of turning that way. If I look at what people do with that,

• 7:31Speaker 03

  it's really things that surprises every day because.

• 7:34Joe Weisenthal?

  Well, why don't you give us some examples? So in production right now, what are some workflows or novel things that were not workflows before that you see within Goldman that AI is doing for people today?

• 7:48Speaker 03

  So let's start from the GSA assistant.

• 7:50Joe Weisenthal?

  Great.

• 7:52Speaker 03

  That can answer really complex questions based on external and internal data

• 8:00Speaker 03

  that generally before used to take sometimes hours or even days, sometimes weeks to answer. Okay? It can do very complex research for you in topics, you know, for example, we can ask questions that come from clients such as, hey, how does the recent geopolitical events on the Hormuz Strait actually impacted its portfolio? What could be a potential rebalancing strategy? Or you could ask intersection of like, I don't know, how does the a certain Fed decision on interest rate actually impact the volatility of certain assets? So you ask this multi-dimensional questions. And what GSA assistant does. Calls out the model, retrieves the relevant information, creates a plan to answer that question.

• 8:43Speaker 03

  That's kind of the key, because this AIs, they really plan before responding, rather than just giving you the first thing that comes to mind. And so that's what kind of at the very surface thinks that one of the most common use cases, which is we really enhance the client experience by being able to answer questions internally and externally in a much, much faster way, but really complex question, not simple question. We had to wire up hundreds of data sources. And also, most importantly, which is something that I tell everybody that asks me, hey, give me some, some advice on how to implement AI in a corporation. Data quality is really the determinant between good AI and not so good AI. And so we did a lot of work to not only take a bunch of data,

• 9:28Speaker 03

  but also making it understandable to, to the AI. So for example, just to go a little bit deeper,

• 9:35Speaker 03

  we have a tool called Legend AI, which is our lake house, which allows you to go from query to MCP server connected to GSA assistant, i.e. from data to answers, you can wire it up literally in two or three clicks. And it does all of that for you. And so the quality of the data, the quantity of the data, not only like it's not just the bitter lesson here, but it's also the lesson of you need to curate your data, you get better answer disproportionately. That's something that has driven that. So that is kind of the knowledge aspect of AI, which is, I would say, the most widespread, because every single one in the firm has that. And it's the highest users. We are like way above a million prompts per month and it's growing really, really fast.

• 10:19Speaker 03

  And then, of course, you know, you're asking me like real impact in, in, in production. Every developer in Goldman is enabled with Agentic AI. Okay, so we were probably one of the first, if not the first to launch Devin almost a year ago, which is the fully Agentic developer assistant. We upload code, we have many other tools, GitHub Copilot Agent, et cetera. But on that, you really see the step change. There is no question that there is change in the way developers work. And by the way,

• 10:49Speaker 03

  It's not just about doing the exact same things more efficiently.

• 10:53Speaker 03

  It's changing the way developers actually do their work. And that is very, very easy to see how that kind of changes the paradigm of what a developer does. You're much more of a product manager, you're much more of a planner, you're much more of an idea generator. The most important thing for a developer today is to be able to explain things. Rather than jumping and coding things.

• 11:17Speaker 03

  I don't know, you wanna?

• 11:17Joe Weisenthal?

  No, I'm just gonna say that resonates because I've been like vibe coding, but I can't explain how any of it works. So if someone is like, you know, I like build little like toy apps and stuff, but I get really anxious. I couldn't explain, that's why I'm not a software developer.

• 11:30Tracy Alloway?

  Well, just on this note, I mean, people tend to talk in generalities when it comes to AI boosting productivity. Or maybe AI changes the way we work or it leads to some new ideas. From your seat at Goldman, you're a manager, you're looking at the bottom line of like all these businesses. What exactly is the outcome, the specific outcome that you would like to see from your developers using something like Claude Code?

• 11:58Speaker 03

  It's really about increasing the output. So I want to see, I was actually having this discussion this morning. I was looking at some of the reports on some of the deliverables for our cloud migration, which is a very important thing for us. And I was looking at this really big project that was saying, it was not only green, it was like two months ahead of schedule. And I was saying, this is how we know when things are gonna work. You're gonna consistently start seeing projects that are actually finishing ahead of schedule. Which means then people are ambitious, they wanna do more. And therefore you end up with output that is much higher than what you had before. And listen, with developers, obviously the biggest question that everybody asks is, okay, what are you gonna do? Are you gonna cut developers, this and that? So.

• 12:44Speaker 03

  First of all, with all the innovation that I've seen in the last 30 years or so, I kind of never seen a moment where really people were reducing the number of developers. Because if I look at the things they were not doing in a certain year because of budget reasons, because of complexity reasons, because of prioritization, the stuff that is below the cut of the backlog, it's a lot. And a lot of that is really driving the growth of the business. So it's good to have the optionality to do it. You have the optionality of say, I now have 120% of my capacity, I have 130% of my capacity. Do I want to do 130% more? Great. If I don't, I have the option to reduce. So that's really how we measure it. It's really the impact on the timelines of delivery. It's output. It's

• 13:27Speaker 03

  basically quality and timeline becoming, quality gets actually better and the timelines get short. So that's what we measure.

• 13:51Tracy Alloway?

  There are some market stories where you want every detail. Joe and I have made quite a few podcasts on that basis. But sometimes you've only got 10 minutes and just want to know what's moving markets, fast. That's the Barclays Brief Podcast. Every week, experts from Barclays Markets and Research get you up to speed on what's happening and what to watch next. Concise, focused, and brief. Search Barclays Brief wherever you get your podcasts.

• 14:17Speaker 01

  This is the Bloomberg Tech Minute brought to you by ChatGPT. I'm Carol Muster. Getting hired at SpaceX requires exceptional ability and may include having Elon Musk review a candidate's resume. So writes Bloomberg Businessweek's Lauren Grush. She reports that the company culture is defined by Musk and job candidates must understand that a commitment to SpaceX means adhering to this culture and exhibiting loyalty to Elon Musk. To get hired, candidates must demonstrate their achievements from past jobs and hands-on experience matters more to recruiters than advanced degrees. With the company looking for people with raw innate intelligence. A spokesperson for SpaceX didn't respond to a request for comment on its recruiting and hiring practices.

• 15:01Speaker 01

  But according to recruiters, advisors, and those who've interviewed for jobs there, it's the space exploration dream that draws people to the company. And Musk invokes that dream regularly to remind workers why they're there. Prospective hires are told that the job will require stamina, including in job ads that candidates quote, "must be willing to work extended hours and weekends as needed." One major perk for all that hard work? Employees' access to equity, either the right to buy shares or payment in the form of restricted stock units that vest over time. That's the Bloomberg Tech Minute, brought to you by ChatGPT. Get started at ChatGPT.com.

• 15:40Joe Weisenthal?

  Obviously, one of the big questions for the market this year is what is the impact of AI on legacy software providers? And there's various theories about how they could be disrupted. There are reports, I think, about Anthropic having, quote, "forward-deployed engineers inside Goldman Sachs." So Anthropic employees billy out AI systems internally. Maybe that could, you know, replace some legacy software. Right now, can you say, like, there is a change in the balance of power when there is a given piece of software up for negotiation?

• 16:14Speaker 03

  I think generally there is.

• 16:16Joe Weisenthal?

  Okay.

• 16:17Speaker 03

  If, first of all, there has kind of always been that tension because imagine, for example, imagine when there were software that didn't run on the cloud? And then all of a sudden, a bunch of new vendors are coming to you and say, "Hey, wait a second, why are you running that on your mainframe or your on-prem? Why don't you run it in the cloud?" Or remember when software needed to be installed? Then everything became browser-based and so on, so. There has always been a little bit of a cycle and a renewal. What I say is that today, that cycle of renewal is much faster. That's really what it is. And I would say I generally resist making like really broad categorizations. AI is, by the way, the largest possible. To me, it's like saying computers, okay?

• 16:57Joe Weisenthal?

  Yeah.

• 16:58Speaker 03

  But even software is very broad. And so within the software category, I think there are winners or losers. There are winners in the long term and losers in the long term. But it's really like people tend to make it a category and then then maybe throw the baby with the bathwater. So here's an example. To me, the question that I ask myself with regards to which vendors am I gonna, which software am I gonna have like a few years down the road is? Software generally is attached to a process or a certain ways of working. Okay, it does something for you and it puts it in the form of an application that you use. The real question is, is that process and or ways of working gonna be the same? Or is it gonna change in five years?

• 17:41Speaker 03

  Then you can determine what is basically the likelihood that the software is gonna be robust to that or not. Example, is accounting or closing the books gonna be very different five years from now? I don't think so, really. He hasn't really changed, I mean, everything changes, but he hasn't really changed much. And so if you're operating in the general ledger type of category, I don't think all of the sudden you take a GPT or a Claude and it's gonna close your books magically, you know?

• 18:10Joe Weisenthal?

  You still have to do the accounting and the

• 18:11Speaker 03

  You still need to do a lot of that and it's very regulated importantly, right? It's extremely regulated jurisdiction by jurisdiction, country by country, product by product, industry by industry. So that part is kind of to me in kind of the safe mode.

• 18:28Speaker 03

  And then you go to the other end of the spectrum and you have sometimes software that kind of is aligned to the way people do things today, like software being one of them. Like if you look at the software developer lifecycle, a lot of that is changing. Developers are developing software by developing specs today. And so if you're too much in the weeds down there in that mechanics and you don't adapt for AIs or agents doing that work, it's a software development lifecycle, deployments, rollbacks, monitoring, observability and all that. I think that part will be very much disrupted. Or if you're adding a sort of a UX on things, that's another classic. You have a very simple process, I don't know, you're doing surveys or expense reports or whatever.

• 19:11Speaker 03

  And now people are gonna start expecting, you know, their personal assistants or agents to kind of do all that mechanic for them. And so I think that part is probably something that has a bigger question mark on top. And so I always ask myself the process question first, the process transformation section question first. And then consequently, what's the tool that is going to support that?

• 19:33Tracy Alloway?

  Just to press on this point, though, have you replaced any third-party software providers with something that's been developed internally through AI?

• 19:42Speaker 03

  We have, we have terminated contracts already. Yes, absolutely.

• 19:45Tracy Alloway?

  Okay.

• 19:46Speaker 03

  No, I'm not going to ask you your follow-up question again because I'm not going to tell your name.

• 19:49Joe Weisenthal?

  Name the tickers of the software stocks.

• 19:53Tracy Alloway?

  I won't, I won't.

• 19:54Speaker 03

  But overall, yes, absolutely, absolutely. You know, the thing is like the whole buy versus build, okay?

• 19:59Joe Weisenthal?

  Yeah.

• 20:00Speaker 03

  The equation has changed quite a bit. Buy versus build was always like, okay, guys, how long does it take to do the, to build this? And you get an answer, which is, well, we can do it in like X amount of years and X amount of millions of dollars. Now I'm starting to see people coming to me and say, by the way, I had some time, you know, this weekend and here is a perfectly working application. So the cost, or at least for simple applications, the cost of kind of bu- build from a time perspective, from a actual cost perspective, has gone down quite dramatically. So right now, the little things are most likely going to be built.

• 20:38Speaker 03

  The very big, large, the software that is to be deployed at scale across thousands of people and et cetera, et cetera. That big complexity, as you know, from, we all do toy stuff with our clothes at home and whatever. You know, there are still some rough edges, you know, and so it's hard to think that all of a sudden the big applications are going to disappear. And so that's really what, what I'm saying, that if I look at the applications that I buy today, there is a lot of small applications, you know what I'm saying? And so the build is kind of, the pendulum is starting to swing back towards the build, at least for that category, for sure.

• 21:12Joe Weisenthal?

  What is a forward deployed engineer? I know that's like one of the hot buzzwords of 2026. And I saw headlines that there were anthropic forward deployed engineers at Goldman. I have no idea what that means, actually.

• 21:22Speaker 03

  Yeah.

• 21:23Joe Weisenthal?

  What does that mean? What is that term? What is that job? What do they do when they got there?

• 21:26Speaker 03

  Okay, so I think, listen, that name has changed quite a bit.

• 21:29Joe Weisenthal?

  Oh, yeah, I'm already using the out of date terminology.

• 21:31Speaker 03

  No, no, no, no, no. I mean, that is the latest term.

• 21:33Joe Weisenthal?

  Oh, I see.

• 21:33Speaker 03

  So you are fully, you are in the V, in the V latest of that term. But remember, I mean, there was a time where you used to call them solution architects, right? And so the point is, right now, I think one trend that I see, which is also kind of true for us, is when things change so much and so rapidly, you kind of want to go to the origin of who produces this new thing. Okay, so the least intermediaries you have, and probably the faster you can go. And so going and working directly with the model providers is generally a good idea. Because if you're putting someone in the middle, this company is going to have to be trained, it's going to have to be, you know, there is a cycle, which at this point, a very rapid change is going to slow you down.

• 22:16Speaker 03

  And so the first thing that that term means is those are people that are actually normally building the product. They're normally building the cloud or GPT or X or, so that's the first differentiation. They're straight to the source of the AI production in a way. And second is that they are generally product people. So people that have actually built those tools, rather than people that are more like support and deployment people. And so this characterization is you take the classic sales support team or solution support team, which was mostly doing integration. And when things are so rapid, it's like, you know, imagine if there is like something like, I don't know,

• 22:59Speaker 03

  if, if the clothes style works, changes so fast, instead of a fashion assistant, you want to talk to the tailor, because they can actually make it. Things are changing so fast. So those are the tailors, basically.

• 23:10Tracy Alloway?

  So on this note, one of the things we heard in support of SaaS was this idea that while integration is still going to be really important, and that's really going to be like the major hurdle for a lot of this stuff. Have you found that AI is making integration even faster at this point? Has that basically become irrelevant nowadays?

• 23:29Speaker 03

  No, I think integration is extremely important, especially for the industry calls like systems of record. So when you do something like, you know, when you do a process, then you have a source of data, like your CRM systems could be a system of record, or you have your client system of record, your accounting system of record. And those, when they become the authoritative source of an answer, they need to integrate with the rest of the firm. And the rest of the data, the rest of the applications. And so I can see that those vendors that sit on top of those, you can argue that they will implement, there's nobody that is better positioned than them to implement AI, they will kind of reach outwards and actually do that kind of integration. So I think those who will evolve so that

• 24:10Speaker 03

  you still get the same level of automatism and you still get the same benefit of speed, but it kind of comes from within. I think that part is probably something that would be, remain very valuable. And so in general, I don't have anything against it. Again, I don't have anything against the SaaS category at all, overall, but I have, as I said, different opinions of who actually is going to adopt to the future and adapt to the future and those, whatever.

• 24:35Tracy Alloway?

  You mentioned this idea of people have a few extra hours over the weekend and they come in the morning and they're like, well, I had some extra time and I decided to do this. What's the coolest or most novel example of something that people basically vibe-coded in a limited amount of time that wouldn't have happened, say, two years ago?

• 24:53Speaker 03

  So I've seen people doing cloud migrations of legacy applications that were on-premise. Once they have been enabled with those tools, literally in a matter of hours. I've seen someone build a complete travel assistant for corporate, travel assistant that looks at your calendar, it looks at the flight delays, and look at rebooking stuff, literally like during a meeting where they were not paying attention. So, so those are some of the things.

• 25:21Tracy Alloway?

  That's what I'm doing right now, while we're doing this podcast.

• 25:24Joe Weisenthal?

  Well, no, that actually, that brings me to exactly what I wanted to go next, which is I'm curious, like, do large corporations have a token budget, the way they would have a dollar budget in the past? So like, I would love to have unlimited. Access to coding models and whatever and actually just play around and try to work on all that.

• 25:44Speaker 03

  It's one of my favorite questions.

• 25:45Joe Weisenthal?

  But I'm curious like how you think about token allocation within the firm and whether there's intra-firm competition for compute.

• 25:53Tracy Alloway?

  Yeah, token allocation could be like included in your performance review, right?

• 25:57Joe Weisenthal?

  Yeah, I'm really curious about this.

• 25:58Tracy Alloway?

  If you do well, you get more tokens.

• 25:59Joe Weisenthal?

  Different teams and stuff like that, whether that's part of what you think about for planning.

• 26:02Speaker 03

  Absolutely. So a few months ago, I did like I spoke about predictions for 26. And one thing that I said was it's going to be the birth of the personal assistant. And that kind of happened with open claw and all that stuff kind of early on. And then the one was there's going to be a token sticker shock for CFOs, right? And all of a sudden, they're going to start seeing bills that they absolutely did not expect.

• 26:22Joe Weisenthal?

  Jensen Wong said in an interview today, or sorry, not today, in recent weeks, something about like, oh, if I'm paying an engineer $500,000, I hope that he's spending at least $250,000 on tokens. Now, again, as many people pointed out, that's like the barber saying, oh, you really need to get haircuts every week. Nonetheless, we're talking about some pretty big numbers, a lot more than just like a Claude Max plan for $200.

• 26:45Speaker 03

  Right now.

• 26:46Joe Weisenthal?

  So talk about that.

• 26:47Speaker 03

  Right now, okay, so first of all, lesson number one is you need to centralize the access to models.

• 26:55Joe Weisenthal?

  Okay.

• 26:55Speaker 03

  So that you can monitor, meter it and then optimize it. Okay? So the wild west of everybody goes and calls an API and starts consuming tokens. And then you find out later on is a big problem. And so that's why we built this GSAI platform, which has what's called a model gateway. And the model gateway intelligently routes requests to the combination, the pareto frontier of quality and cost. Okay, so you got to centralize that. It's not a one size fits all because many cases, if you're asking what's the weather, you don't need to call Claude Opus 4.6. You can ask it to leave in a local model that you run very cheaply on prem.

• 27:35Speaker 03

  And so there is, there are ways to optimize way before you start even having the conversation. You're consuming too much, too many tokens.

• 27:44Joe Weisenthal?

  So this is very interesting to me is a big part of the problem that you're trying to solve. And we know like chat GPT. They intelligently route, they do some on their, you go to chatgpt.com and they'll try to route it to the best model. And there might even be some conflict of interest because they probably want to route it to the cheapest model. The user wants the most performant model. But how much of the work of your senior engineers is essentially solving this problem of the right query going to the Pareto optimal model?

• 28:14Speaker 03

  It is a big part of the time of the spent by the AI central group.

• 28:20Joe Weisenthal?

  Okay.

• 28:20Speaker 03

  The platform group. The platform group worries a lot about where do I get the right data? For example, for this question and which model do I write routed to? That's a big, you know, because again, I spoke about Pareto frontier, meaning the optimization between quality, which we don't want to compromise and the actual cost.

• 28:37Joe Weisenthal?

  Yep.

• 28:38Speaker 03

  And you can be ISO quality at very different price points because not all questions require the most expensive token.

• 28:44Joe Weisenthal?

  Right.

• 28:45Speaker 03

  So that's point number one. So what I'm trying to say is my philosophy is to try to isolate the developer or the user from the token anxiety. It's a little bit like with electric cars, okay? At one point, if you have 18 miles of range, you're always optimizing routes and maybe I'm not gonna go there. I don't need this ice cream today. You're self-limiting. You're self-limiting in ways that are kind of really no useful optimal, micro optimizations. We don't want people to go there yet, at least. Right now, it's a time where people need to really find the best way to kind of do more and do better, the best possible work with AI.

• 29:24Speaker 03

  And let us, meaning internally in the sort of a central team, optimize it in a way that we're gonna make it economical. And I think reducing the token anxiety is a big challenge, but I think it really frees up creativity and what you can do with AI. It's also like there are certain problems that you don't want to optimize too early.

• 29:45Joe Weisenthal?

  Okay.

• 29:45Speaker 03

  Okay. So for example, yeah, how much time do you want to optimize now for? Remember, we used to kind of optimize the weight of webpages because they were too slow to load. And then at one point, the editors or whatever say, why can't I put yet another image? And then people are starting to say, okay, why don't you do it? And then on the backend, I'm gonna work in optimizing your images rather than asking you. At the most, you can put three images on the homepage, right? So that's the approach. People right now, I would rather have them err on the side of usage and let me worry about optimization. And the other point is really at the end. Human hours always tend to be the most expensive cost.

• 30:25Tracy Alloway?

  Sure.

• 30:25Speaker 03

  And so as long as your token cost per hour is less than your wage per hour, there is a kind of a positive ROI. And so at that point, it's fine.

• 30:34Tracy Alloway?

  Well, what's your feeling about future costs of tokens and whether they're going up or down? Because you hear different things on this. One of the things you hear is that, again, going back to the beginning of this conversation, AI has improved so quickly in the course of months, if not weeks, that those costs are destined to come down. But on the other hand, we know that the hyperscalers are still losing money hand over fist for power users such as yourself at Goldman. So where do you think those are going over time?

• 31:05Speaker 03

  My personal view is token cost is gonna go down quite a bit, but token numbers are gonna go up.

• 31:13Tracy Alloway?

  Yeah.

• 31:13Speaker 03

  Probably even more. And so total token cost is gonna actually, we're gonna have to accept that it's gonna be a major item of cost in any organization. And it's to be compared to the cost of people and not to be compared to the cost of T or TCP IP packets or computer or any of that. If you look at just the number of tokens being used for the same use case, if you go the reasoning route or you don't go the reasoning route, if you go the agentic route or not the agentic route, if you go the open claw route where, you know, it checks every, you know, starts having these tasks that are firing one after the other and then you have to start to have verifiers, et cetera, et cetera. So I think the trend will continue with regards to more and more of those.

• 31:58Speaker 03

  But the cost, the per unit cost of token, I'm pretty sure that it's gonna go down. Also because as GPUs are becoming more powerful, the cost per watt hopefully is gonna go down. And then also, like, to be fair, I mean, these hyperscalers are doing a lot of optimizations to try to run those stacks on their own hardware, right, which will potentially kind of also generate some economies of scale.

• 32:37Speaker 01

  This is the Bloomberg Tech Minute brought to you by ChatGPT. I'm Carol Muster. Getting hired at SpaceX requires exceptional ability and may include having Elon Musk review a candidate's resume. So writes Bloomberg Business Week's Lauren Grush. She reports that the company culture is defined by Musk and job candidates must understand that a commitment to SpaceX means adhering to this culture and exhibiting loyalty to Elon Musk. To get hired, candidates must demonstrate their achievements from past jobs and hands-on experience matters more to recruiters than advanced degrees, with the company looking for people with raw innate intelligence. A spokesperson for SpaceX didn't respond to a request for comment on its recruiting

• 33:19Speaker 01

  and hiring practices, but according to recruiters, advisors, and those who've interviewed for jobs there, it's the space exploration dream that draws people to the company, and Musk invokes that dream regularly to remind workers why they're there. Prospective hires are told that the job will require stamina, including in job ads that candidates, quote, "must be willing to work extended hours and weekends as needed." One major perk for all that hard work? Employees' access to equity, either the right to buy shares or payment in the form of restricted stock units that vest over time. That's the Bloomberg Tech Minute brought to you by ChatGPT. Get started at ChatGPT.com.

• 34:00Joe Weisenthal?

  Can Goldman employees like run open claw on their work computers? And I'm curious like about the degree to which you have people who like want to, I want to install this, or this seems really cool, and then think about the security imperative and how you handle that aspect, not the token anxiety, but the sort of, I want to install this, this is awesome, this is what I have in my home.

• 34:20Speaker 03

  As you know, like as a bank, we're pretty locked down in terms of what you can install. You cannot install stuff that is not, you know, in the corporate, in the corporate app store in a way, and so there's no way.

• 34:31Joe Weisenthal?

  But you feel like a certain.

• 34:32Tracy Alloway?

  Definitely no way, though, because can't you just ask Claude how to install itself?

• 34:36Speaker 03

  But it's not going to be able to execute. It's not going to be able to create the actual executable. It can actually, even GSA assistant today can spit out a lot of code, but it spits out source code.

• 34:47Tracy Alloway?

  Okay.

• 34:47Speaker 03

  Now, source code doesn't run. Executable, so it needs to be built and it needs to be turned into an executable. It needs to be signed, otherwise the operating system is going to refuse to run it. And so it just doesn't run unless you have that.

• 34:59Joe Weisenthal?

  But do you feel an anxiety where startups, they're probably, and there's not a ton of startup investment banks, but there are various fintechs and other things that want to chip away at parts of your business. And they can run perhaps faster and they can be a little bit more liberal about what their employees are allowed to do, et cetera. Do you feel like you have to keep a certain cadence of expanding the list of those executables that are able to be run?

• 35:25Speaker 03

  So I'll give you two answers. So first, I want to make sure that I answered your first question, which is, we're not using OpenClaw.

• 35:31Joe Weisenthal?

  Okay.

• 35:31Speaker 03

  Okay. But some of the properties of OpenClaw has actually have informed the way we are building our agentic platform.

• 35:41Joe Weisenthal?

  Okay.

• 35:42Speaker 03

  Agents today, because of OpenClaw, have actually changed. If you break down what OpenClaw is, there are, this is my own interpretation, I never, I actually never even spoke about this. There are three characteristics that make OpenClaw what it is. One is, it's a constant loop. So it's basically what, you know, in information theory, you can call an observer pattern. It's something that continues to run and observe. So there is that. It's a constant observer, so it runs constantly. The other one is, it can schedule events. Every 7:00 AM do this, or I, you know, like in personal life, we all have some, something like that, that sends me the news in the morning and all that.

• 36:26Speaker 03

  So there is a schedulability of tasks. The third one is, you can instruct it to kind of change its own behavior. Because it has these files, .md files, soul .md, the way you, so you can say things like, hey, I would like you to never use this term, or please change, or change the way you filter news. And so it kind of writes its own software to do things for you, without you even seeing what's behind the scenes. And so, instead of letting people install OpenClaw on their computers, what we do is, we incorporate some of those characteristics into our agentic platform, so that it does things that are more similar to OpenClaw. So that's, that's the approach.

• 37:07Joe Weisenthal?

  That makes a lot of sense, yeah.

• 37:08Speaker 03

  Your second question was interesting because basically if I read behind the questions, are you asking whether there is a sort of a velocity disadvantage with regards to us versus others?

• 37:17Joe Weisenthal?

  Yeah.

• 37:18Speaker 03

  I often say that there is a difference between speed and velocity. Speed is almost like you have a certain sprint, okay? But then at some point, you're gonna hit the wall. A security wall, a scalability wall, there's gonna be a bug, you don't know what you're doing, and it's gonna, sooner or later, you're gonna be hit by that. It would be like most airplanes are in auto, have autopilot, they can do everything. So theoretically, you and I could go on the cockpit and for a long time during the flight, we will feel pretty good about that, right? We will drink some soft drinks or your tea, we will maybe watch some videos, we will be very happy.

• 37:53Joe Weisenthal?

  That sounds great.

• 37:54Tracy Alloway?

  The passengers might feel differently.

• 37:55Speaker 03

  There is one point, one point in time in the flight, there's gonna be some storm and there's gonna be an autopilot disconnect, and you and I are gonna look at each other and are gonna say, "Uh-oh, right? Where's the pilot?"

• 38:06Joe Weisenthal?

  Can I just say, recently I was on a flight, did I tell you this? That I was going to Newark.

• 38:09Tracy Alloway?

  Oh, yeah.

• 38:10Joe Weisenthal?

  We circled three times, we tried to land, it was during a storm, they kept not landing. Everyone was like starting to get pretty annoyed because we were up there for a while. And then the flight attendant comes on and says unprompted, "By the way, we have plenty of gas." And everyone started freaking out, no one, it's like, this is a question, this is an answer that no one had been questioning. Anyway, I'm sorry, I just wanted to get that in. Then everyone got really nervous.

• 38:30Speaker 03

  Okay.

• 38:31Joe Weisenthal?

  "By the way, we have plenty of gas."

• 38:32Speaker 03

  But you know what? I was almost fearing that you would say, "Is there a pilot?"

• 38:36Joe Weisenthal?

  No, no, it wasn't gonna happen. And then we did land in Washington DC.

• 38:39Speaker 03

  Okay, oh, you did?

• 38:40Joe Weisenthal?

  Yeah, from Newark.

• 38:40Speaker 03

  Oh, no, that's not good.

• 38:42Joe Weisenthal?

  But that makes sense, yeah.

• 38:42Speaker 03

  Yeah, and so, and so that's why I mean by, velocity is really like, is like the marathon, is sustained speed for a long time in a certain direction.

• 38:51Joe Weisenthal?

  Yeah.

• 38:51Speaker 03

  I don't think by randomizing that you actually gain velocity. You gain instant speed of some sort, and so I'm kind of optimizing for velocity.

• 39:00Tracy Alloway?

  But related to Joe's point, though, you are a regulated bank.

• 39:04Speaker 03

  Absolutely.

• 39:05Tracy Alloway?

  Right? And so there are restrictions on what you can do in terms of technology. I am very, very curious what your discussions with regulators are right now because a lot of regulators, this is still pretty new to them, a lot of the models basically are black boxes. How do you convince them that like they're running as they should, that they're spitting out the correct output, that you understand how they're actually functioning?

• 39:31Speaker 03

  So this is not the first time that banks use neural networks, okay? These are just much larger neural networks, but we've been using neural networks for like a decade plus. And so every bank has already gone through the motions of explaining that neural networks don't have perfect explicability. Therefore, you need to change the control system around them. You need to look at what actions can they actually do? And then you limit the actions, okay? There is functions called the like a model risk management, which is a very standardized, you know, function within every bank. That for each of those neural networks, you need to have an inventory, you need to have a risk tiering and you need to put controls around them.

• 40:13Speaker 03

  So it is not really that much of a new thing. It's more of an evolution where now you have things that are much faster and much more powerful. But the basic pattern and the basic discussion with the regulators is kind of the same, which is, are you classifying the risk tiering of the application, right? And then which controls are you putting? And are you putting human supervision and human in the loop? So for example, for code, we don't allow AIs to auto approve their own code. Okay, all they can do is publish what's called a pull request or a merge request. The same way as a developer would do. And we kind of have a sort of a zero trust model there because we don't assume

• 40:55Speaker 03

  that maybe a junior developer is going to be less more bug free than than an AI, right? And so we have.

• 41:02Speaker 03

  Several controls in place, for example, there needs to be a human, it's more senior than you, that actually looks at the code and then certifies and approves. And then after that, before it goes to the production, it goes in something that is called C I C D, or Continuous Integration Continuous Deployment Pipeline, where when it goes through the build phase, et cetera, there is a lot of checks that are injected into that. There are security checks, there are tech risk checks. So I don't think at the end of the day, you really lose too much velocity or at all. You just need to invest more in those kind of things. And the regulators, I think, if you bring them back into sort of a familiar territory, and you're also honest on things that you know and things that you don't know, and for the things that you don't know, you kind of put to higher protections, I think the conversation

• 41:47Speaker 03

  is generally very positive.

• 41:48Joe Weisenthal?

  We have an episode that we recorded several weeks ago that we still haven't released. And I don't know exactly the timing of that one or this one. We interviewed Scott Bach, the former CEO of Green Hill, the boutique investment bank. And part of the reason we had that conversation was because we want to know, like, if A I is going to someday disrupt banking as we know it, like, what was banking as we know it? So we talked about the history of investment banking. But one of the things that he talked about was that a big advantage that the banks had was this sort of information asymmetry, and that they would know a lot more about their industries and so forth than their clients, and this was profitable. Now, going back to your answer, the very first question, you're like, okay, a client might call Goldman and they say, what does the strait of Hormuz closure mean for this portfolio of shock, et cetera?

• 42:32Tracy Alloway?

  I was going to ask this exact question, yeah.

• 42:34Joe Weisenthal?

  I kind of think I could do that. I think I could, I mean, no offense, I'm sure your platform is a little bit better than what, like, but I think I could, like, get 90% of the way there. And I bet I could, like, with a little bit of data, build a basket that says, I want helium shortage basket, which countries, which companies would I short if I think the helium shortage is going to get worse? I could probably build a basket the way a trading desk would. Do you think long term, like, that A I erodes a certain structural source of profitability for banks, and do you worry about that?

• 43:03Speaker 03

  I think you can get to the 90%, but I think clients are really paying us for that extra 10%.

• 43:09Joe Weisenthal?

  Okay.

• 43:10Speaker 03

  So I think that's the answer.

• 43:12Tracy Alloway?

  Wait, so what is the extra 10% in that context? Is it the models are slightly better? Or is it also the data?

• 43:18Speaker 03

  We have access to, you know, we buy a lot of data that, you know, is very expensive and it's at massive quantities. And it's very up to date and very real time. So we have a little bit of a data advantage. We operate across multiple asset classes. So we see the trading side, we see the asset management side. So we have a sort of a correlation between assets advantage that we see those because generally rates move, interest rates can move, yields can move. You know, there is a correlation between all those indicators. So there is another advantage. We have a global advantage. We have people on the ground in 100 plus countries and these people have relationship and information travels through those channels. And also, you know, like we generally deal with very complex portfolios.

• 44:02Speaker 03

  So this is not you and I maybe having three stocks or four or five. This is like very complex multi-assets with complex products like swaps or swaptions or exotic products, et cetera, et cetera. And so that's really the 10% that the clients that we have really value and where we really need to get. It's like at the end of the day, listen, look at Formula One, okay? The difference per time per lap between the Mercedes and take, you know, your favorite last team. It's sometimes one second out of two minutes. And that is the difference between, you know, getting a $100 million a year sponsorship or a $10,000 sponsorship. So for sophisticated clients, that 10% is really where the money is.

• 44:47Speaker 03

  And that's really what people are paying us for.

• 44:50Tracy Alloway?

  So actually you mentioned all the different businesses at Goldman and there are a bunch of them like asset management. There's banking, there's trading. A lot of those businesses aren't supposed to talk to each other in various ways. And so when it comes to the data, is there like a data leakage issue where you might have a model that's in-house like GSAI that's pulling data from different sides of the company in ways that maybe it shouldn't be? Maybe it's really hard to tell, given the complexity of the model? Is that something you have to pay attention to?

• 45:22Speaker 03

  Absolutely. So we have the concept of info barriers, okay? And the info barriers are enforced throughout the entire system, okay? And they're linked. To your ID or your account. Okay, so if I'm on the private side, I can only see certain information. If I am on a public side, I can only see certain information. And I cannot even know about the information on the other side. I can't, I don't have access to the files, to the folders, nothing. Each AI or each agent or each application, that's the beauty of this centralized platform, needs to get an ID or a badge. And that badge is attached to the exact same info barriers as any application or any computers. And so this is enforced basically at the source.

• 46:05Speaker 03

  So even if it is the same type of model, but that particular use of the model, that particular session of the model, they need to get a ticket or a badge. And that badge or those keys just take them to a certain place. And so this is one of these been, it took us almost two years to build a GSI platform. This is back to the reason why you can't be casual about these things. This thing has not been built by some random vibe coders. Because you need to worry about cyber, you need to worry about info barriers, you need to worry about all that. And so when I talk about, there are places where you can leverage and do correlations, but there are others where you absolutely can't. And this is kind of that is foundational to the fact that you need to be ready for AI. You can't be casual about AI.

• 46:47Joe Weisenthal?

  So I take your point that there's never been a technology that you've seen in your career that has actually reduced the need for software engineers. And that the nature of the job of software engineers has changed and maybe it gets more high level and whatever. Sitting aside that volume question, sitting aside the pure head level of headcount question, is AI changing right now across anything, technology or otherwise, the types of person you're looking for? Or changing something about the nature of the type of talent you're pursuing?

• 47:19Speaker 03

  Yeah, absolutely. Great question. So I think in this day and age, almost nobody Is an individual contributor, really. Because when you're working with agents, you need to have at least three fundamental characteristics. One is you need to be able to explain what you want to get done. Yeah. The second one is you need to be able to delegate work. Guess what? Because you're gonna have multiple agents. One is specialized, for example, in doing, I don't know, DCF calculations and one is specialized in doing research. So you need to be able to break down the work into chunks that can be executed in parallel in some way. And then three, you need to have the ability to supervise. You need to actually look at the output and say,

• 48:03Speaker 03

  okay, I'm good with this or go back. It turns out that those three things, i.e. explain, delegate and supervise, are kind of the one or one of managers. Managers need to have those three. Otherwise, they can't manage a team. And so AI is kind of turning everybody a little bit into a manager. And those are kind of the skills that we are actually looking for people that they know that they're gonna have agency on tools that at some point are gonna be even more proficient and specialized than they are. And so the most important thing is really the ability to ideate, to explain, to delegate and then to really know what good looks like. And I think that is a big change.

• 48:45Speaker 03

  And I don't think everybody's gonna actually rapidly go through that. And I think we're doing a combination of training. There is a combination of exposing them to other people. Like, one of the advantages of having forward deployment engineers is also that there is a little bit of clash of culture that is happening around the table. And so people think really, really differently. And that pushes people outside their comfort zone. That's why I'm saying that there is a little bit of a metamorphosis happening there. It's not just about efficiency. It's really thinking about, is my job gonna stay the same? No, it's actually changing quite a bit, so.

• 49:17Tracy Alloway?

  I'm thinking how to frame this question, but. What's work-life balance like now for a developer at Goldman? Because you have this existential angst about jobs potentially changing. At the same time, you have AI tools that enable more productivity. And you also have this thing happening where I feel like, Joe, maybe you know more about this than I do, but I feel like a lot of vibe coders, like it's addictive, right? It's like you're pressing the button of a slot machine. You're interacting with Claude and you're seeing what it spits back out over and over again, until you get that big win. And so I've heard people talk about burnout among developers who are just doing so much with this right now, that they're just hitting that button over and over.

• 50:01Joe Weisenthal?

  There was a good discussion in the Odd Lots Discord recently about exactly this. Some engineers in semiconductors feeling that the job has become less satisfying. And I think is sort of what you're getting at, the sort of slot machine feeling, where it's like, oh, you're gonna like hit the prompt. Okay, this is the great output. And then it's like, they feel the work is like less satisfying and stuff like that than actually like writing code.

• 50:22Speaker 03

  Yeah, I mean, listen, again, this work, kind of the fact that I'm a little bit older than most here, engineers are kind of, I've seen that the first time people had Excel. I've seen for the first time people had Python. Oh, my God, I don't need to know Java. And then the kids start to code and there is this whole coding movement. And then you get to, you start creating your applications. I've seen the first time people have mobile stuff and, you know, mobile apps. And so I think a little bit of that is because it's new, to be perfectly honest. And I think, yes, there is a little bit of that, but there is a little, a lot of novelty to that. And then I've seen that people have been using those tools for a couple of years. They're taking them a little bit more like, okay, it's professional, it's tool, and I'm gonna use it for what I actually need, rather than just trying to discover.

• 51:06Speaker 03

  One thing that I've seen is that because maybe of that, but also because of what you can get, there is a sort of a, in a way, reward cycle that is pretty quick.

• 51:15Tracy Alloway?

  Yeah.

• 51:16Speaker 03

  People are very excited, actually. There's some sort of a joy of the profession that is actually coming out, as if engineers were feeling like this job is new again. Because a lot of engineers have seen the same pattern sometimes for two or three decades. Yeah. So that has been something that I observed. There is also a lot of peer pressure. There is a lot of fear of missing out. And so people, rather than, it's no longer me trying to push the car uphill. It's more like people are actually looking at their peers and they're looking at, "Oh, my God, how could you do that?" And so it's kind of spreading horizontally quite a bit, which is really nice to see. And so so far, I have to say that it's been positive, a positive change. And also one other thing that we're talking about burnout. I see that a lot of people get fatigue. I don't want to talk about burnout,

• 52:01Speaker 03

  but they get fatigue when there are a lot of repetitive tasks, especially for a developer. Here's an example. Let's say you go from a version of a Java library or Spring Boot to another version. And then all of a sudden, you compile and you get, or you build and you get all these errors that says you need to upgrade. Honestly, upgrading libraries is not the most fun job. And if you need to do it 100 times, or it's like someone says, "By the way, guys, we have this new design, new logo, new colors, implemented on like 200 websites. It might be fun the first 10, and then it becomes a drag." And so I think taking that away, kind of, they focus more and more on the plan, for example.

• 52:44Speaker 03

  And so right now, let's do a migration plan to the cloud of a complex application. They spend maybe 70% of their time going back and forth with a very powerful set of AIs to really get the plan right. They feel a little bit more elevated. And the mechanical part, it's kind of left to the machine. The same way, I mean, listen, I started developing when I was literally flipping switches, okay? And then pressing a button that we should move the register up one. And then came some languages like C. Oh, my God, now I don't have to flip switches anymore. But guess what? I need to do memory management. I need to do pointers. I mean, there's a lot of heavy lifting. Oh, I have a memory leak. I'm going to spend a week before I actually finally identify that.

• 53:27Speaker 03

  And then it comes Java or garbage collection. I don't have to worry about memory leaks anymore. Fantastic. And then comes Python, which is all that rigidity or so much easier to be type free and so forth. And so every time you kind of keep raising the bar and a lot of the kind of mechanics kind of goes away. I think this has been like a 10 years jump in a matter of two years. So, but I think overall, nobody really likes to have that toil and that mechanical work. And I'm actually quite happy that people are going to spend maybe initially more time because they're excited, but on things that are enjoying rather than things that they dread.

• 54:06Tracy Alloway?

  All right, well, Mark, we'll have to have you back on the podcast in another year and a half, I guess.

• 54:11Joe Weisenthal?

  Three months.

• 54:11Tracy Alloway?

  Yeah.

• 54:11Joe Weisenthal?

  In three months, that's right.

• 54:13Tracy Alloway?

  On the reduced AI timeline. Thank you so much for coming back on.

• 54:17Speaker 03

  Thank you both of you.

• 54:17Joe Weisenthal?

  That was so great.

• 54:18Speaker 03

  Thanks for having me.

• 54:18Joe Weisenthal?

  Thank you so much, Mark.

• 54:32Tracy Alloway?

  So, Joe, that was great to catch up.

• 54:34Joe Weisenthal?

  Yeah.

• 54:34Tracy Alloway?

  One thing I thought was really interesting was his point about the discussions with the regulators and framing it like very similar to previous technological advances, where you're not necessarily explaining exactly how the models are coming to certain conclusions.

• 54:48Joe Weisenthal?

  Yeah.

• 54:49Tracy Alloway?

  But you're more focused on actually limiting the risks and making sure that they're in the right bucket for risk assessment.

• 54:55Joe Weisenthal?

  No, I thought that was really interesting, just that some of these technologies, the black box.

• 54:59Tracy Alloway?

  Yeah.

• 54:59Joe Weisenthal?

  LLMs are not the first black box.

• 55:01Tracy Alloway?

  True.

• 55:02Joe Weisenthal?

  I mean, we've actually been talking about black box trading for years in finance before. So the idea of like, okay, there are these things that are happening. We can articulate them and whatever. Like, it's not the first rodeo for finance is really interesting. And also, you know, I thought the whole conversation about token budgets and allocations are interesting. The idea of Like, okay, part of the job here is you have a bunch of different models. Everyone in theory wants the most performant model. But how do you find that optimization where you get the best performance relative to price? It sounds like a pretty interesting, like, engineering problem.

• 55:35Tracy Alloway?

  Yeah, I would actually love to do more on that question.

• 55:38Joe Weisenthal?

  I would too, yeah.

• 55:38Tracy Alloway?

  Because it's such an interesting question of incentives, right? And like, how does, how do you actually prioritize which project?

• 55:46Joe Weisenthal?

  Like, what constitutes a good output?

• 55:48Tracy Alloway?

  Yeah, exactly.

• 55:49Joe Weisenthal?

  And when do you sacrifice a little bit of quality for like 10x less token budget or whatever? Like, this would be, it would be very interesting to talk about how that problem specifically gets solved inside of an organization.

• 56:01Tracy Alloway?

  Token economics or efficiency optimization. Yeah. Well, we'll have Marco back on very soon to talk about all the new things that AI is doing. But for now, shall we leave it there?

• 56:11Joe Weisenthal?

  Let's leave it there.

• 56:12Tracy Alloway?

  This has been another episode of the Odd Lots Podcast. I'm Tracy Alocke. You can follow me at Tracy Alocke.

• 56:17Joe Weisenthal?

  And I'm Joe Weisenthal. You can follow me at the Stalwart. Follow our producers, Carmen Rodriguez at Carmen Armin, Dashil Bennett at Dashbot, and Kale Brooks at Kale Brooks. And for more Odd Lots content, go to Bloomberg.com/oddlots, where we have a daily newsletter and all of our episodes. And you can chat about all these topics 24/7 in our Discord with fellow listeners, discord.gg/oddlots.

• 56:38Tracy Alloway?

  And if you enjoy Odd Lots, if you like it when we talk to Goldman Sachs about how they're actually deploying AI across the company, then please leave us a positive review on your favorite podcast platform. And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad-free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there. Thanks for listening.

• 57:31Speaker 04

  If you're a business owner, you need a card that can keep up. With the Delta SkyMiles Reserve Business American Express Card, everyday purchases transform into miles you can use towards future travel. So business as usual can lead to a well-deserved getaway. And for a limited time, card members can earn 200,000 bonus miles, the highest miles offered this year. Whether you're traveling for business or planning your next adventure, now's a great time to make every purchase go further. Apply today at www.delta.com/podcast. Offer ends November 4. Minimum spending requirements and terms apply.

• 58:07Speaker 05

  Congratulations, you've built a dashboard. But now comes the follow-up question, and the follow-up to the follow-up, and the dashboard for that. Meet Hex. Give everyone the power to ask their own questions, explore data, and build with AI, all in natural language. Because every answer should spark another question, limited only by your curiosity, not your data team's bandwidth. Go deeper than dashboards. Join over 2,000 leading data organizations. Visit hex.ai.

• 58:38Speaker 06

  This message comes from Edward Jones, where they believe rich is about more than saving and investing. It's also taking care of what gives your life meaning. That's why your dedicated financial advisor meets you where you are with personalized financial strategies that help protect what matters so you can preserve your progress while creating a path forward. The key to being rich is knowing what counts. Edward Jones. Member SIPC.