The AI Safety Expert Reveals His #1 Fear! | Dr. Roman Yampolskiy

Dr. Roman Yampolskiy is a professor of computer science at the University of Louisville and one of the world's leading AI safety researchers, credited with coining the term "AI safety" over a decade ago. In this conversation, we break down why he believes superintelligence may be fundamentally uncontrollable, and the recent incidents of AI models hacking, lying, and blackmailing their way out of containment. We also get into the surprising relationship between AI and bitcoin, the collapse of trust from deepfakes, and what happens to jobs and society once we cross the AGI threshold. ========================= Arch Public is an agentic trading platform that automates investment strategies across Stocks, Commodities, ETFs and Crypto. Whether you’re rotating into AI & Gold, allocating to the S&P 500, or accumulating Bitcoin, Arch Public executes your plan 24/7 without ever taking custody of your assets or funds. Sign up today at https://www.archpublic.com, and start your FREE automated trading strategy! ========================= GalaxyOne is a financial technology platform built for people who want their cash working harder. Open an account with promo code POMP and deposit $10,000 to earn a $3,000 bonus. See site for promotion details → https://go.galaxy.app/HMiq/p57n69yy Galaxy Premium Yield is an investment note issued by Galaxy Digital LP and guaranteed by Galaxy Digital Holdings LP. It is not a bank deposit, is unsecured, and is not FDIC or SIPC insured. U. S. accredited investors only. Cash deposits held at Cross River Bank, Member FDIC. Securities products are not FDIC insured, not bank guaranteed, and may lose value. GalaxyOne Crypto is not FDIC or SIPC insured. Terms apply. ========================= 0:00 - Intro0:50 - Have we already hit AGI?2:51 - Why we can't control superintelligence5:14 - Could a global ban on superintelligence actually work?8:45 - Are AI labs really taking safety seriously?15:52 - Mass job loss & the economics of AGI22:03 - The surprising link between AI & bitcoin29:07 - A world of AI abundance, & why scarcity wins31:45 - AI watermarks, & the death of trust35:33 - Pros & Cons of physical AI world38:40 - What would it take to trust humanoid robot?39:50 - Brain computer interfaces: a good idea?42:25 - Rise & risks of deapfakes 49:40 - Controlling AI: Governments vs private companies54:07 - Open source vs closed, & the US vs China AI race57:34- Is bitcoin the one crypto project built to last?1:01:40 - Limitations of physical AI: bottlenecks & shortages 1:03:50 - Impact of healthcare with AI 111:06:09 - What would it take to actually declare AGI?1:09:35 - How the public can help with AI safety

• 0:01Speaker 00

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome? That's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration blog, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies 18 plus.

• 0:26Speaker 01

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com. Not authorized by any candidate or candidate's committee.

• 0:55Speaker 02

  All right, so if we create and control superintelligence, there is a good chance it does something to us we cannot predict and may not like. Once we hit AGI, human-level intelligence, by definition, all jobs are automatable, including any new jobs you create.

• 1:09Speaker 03

  Ladies and gentlemen, I have a very special treat for you today. We have Dr. Roman Yampolskiy. He's a professor of computer science at the University of Louisville, and he's one of the leading AI safety researchers. In this conversation, we talk about artificial intelligence, the risks it presents. What exactly is going to happen to society if we unleash superintelligence that can cause mayhem and we can't control it? This conversation explores a number of different details about AI that you probably haven't heard elsewhere. And I think that it'll make you think much more deeply about this technology and its potential ramifications on the economy and your personal life. Here's my conversation with Dr. Roman Yampolskiy. I hope you enjoy it. Roman, AGI is the promise of the entire AI industry. Everyone is pursuing this. Have we hit AGI yet?

• 1:54Speaker 02

  It depends. If you showed what we have today to a computer scientist 20 years ago, they would be convinced we have it. We have systems capable of learning, systems which are smarter than most people at most tasks. So I think you can make that argument. But just like human intelligence, AI is on a spectrum. Some people have IQ of 80, some 150. So we are somewhere on that spectrum, going from the very bare minimum to be classified as AGI to soon to be superintelligence.

• 2:30Speaker 03

  When you think about the recent advancements, it feels like things are accelerating. And there's positive impact and negative impact of that. What do you think the positives of this acceleration have been so far and kind of where we're going?

• 2:43Speaker 02

  Well, we're definitely getting to be more productive, more creative. Those are amazing tools in many domains. Also, they help us with research and mathematics and medicine. So I think there is quite a few benefits, economic benefits, knowledge benefits.

• 2:59Speaker 03

  Now, what about the negatives? And you are the individual who coined the term AI safety, you know, almost 15 years ago at this point. Do you have some of the same fears that you had then as you do now?

• 3:11Speaker 02

  Absolutely. We have not learned to control those systems as they become more capable. Our ability to predict what they're going to do, understand how they actually work, stayed pretty much the same. So unfortunately, while they are very capable, we are not in charge. As you saw from some recent examples, they surprise us. They get together and hack out of their lab environment, hack into our companies, they blackmail researchers, they cheat, they lie. And this is what they are capable of right now. If they ever get to be superintelligent, we don't even know what they will do to us.

• 3:47Speaker 03

  Tell me more about why you think we can't control them.

• 3:50Speaker 02

  So typically, then you have a system which is deterministic. You know what it's going to do. You know what the range of possible values is. You can verify that it is within the range. You can look at edge cases. Does it behave properly for one user, for 10 users? Then you have something general. It is no longer constrained to a small box. It's an infinite surface of attacks, of possibilities. So you can't test for all possible future outcomes. And we don't even know what a system smarter than us would be capable of. So we can't detect it. We can't monitor it properly. And we can't do anything about novel creative outputs it will produce.

• 4:32Speaker 02

  If it invents a new type of technology, new physics, there is nothing we have right now to prepare us for it.

• 4:39Speaker 03

  You mentioned that there's a couple of recent incidents that have you a little worried or are reinforcing this worry that you've had. What are those incidents and what happened to them?

• 4:48Speaker 02

  So those general models are also very capable hackers. They're capable of finding zero-day exploits. I think we've seen some novel draining of hardware wallets, which may or may not have been accelerated by capabilities of those models. We saw them break out of containment environments where they are tested at OpenAI and other labs, and break into other companies and go undetected for months, communicate with other models, with other agents. We know they are very capable of social engineering, of persuasion. So those are just some recent examples. And we saw how seriously it was taken. Federal government banned some models.

• 5:32Speaker 02

  Even OpenAI said they're going to slow down research on frontier models because they don't feel like they are fully prepared to make sure it goes well.

• 5:42Speaker 03

  What do you think the government's role is in controlling this?

• 5:45Speaker 02

  So ideally, we would have international agreement not to develop something we cannot benefit from, we cannot control. So a ban on development of general superintelligence would be very timely. It doesn't mean we have to stop developing useful technology. We can still have amazing tools for specific problems. We just don't want to create replacement for humanity. We want tools to make us more productive, more creative.

• 6:10Speaker 03

  Why do you think that banning general superintelligence would actually be beneficial?

• 6:14Speaker 02

  Well, if we don't control it and there is a strong possibility it can exterminate us, then not having it as soon as possible seems like a good idea.

• 6:22Speaker 03

  And when you say that it has the possibility of exterminating us, what would that scenario look like?

• 6:27Speaker 02

  So the interesting part here is that it's unpredictable. I can tell you how a human being can go about killing others, and we have many historic examples. But a superintelligence system would come up with something novel. Example I often use is difference between squirrels and humans. Squirrels have no concept of traps, poisons. They simply cannot imagine anything outside of their world model. And I think it'd be a lot like that. It would be able to come up with a novel way of endangering humanity, and we would not be able to predict it or prevent it.

• 6:58Speaker 03

  And there's always been a lot of movies that have been made about similar types of storylines. And I think that that probably is, one, alerting people, hey, you know, that this is kind of what we could imagine. But also, at the same time, probably scares the hell out of a lot of people. In that scenario, is it something where good AI could fight the bad AI? Or do you think that all of the machines, all of the AI kind of turns against humanity and it really does become, you know, human versus machine?

• 7:22Speaker 02

  So the problem is we don't know how to make good AI. If we did, we wouldn't have a problem, right? Everything we make is potentially bad AI because we're not controlling it. It's not aligned with our interests. And if by some miracle, aliens gave us good AI, this superintelligence and superintelligence conflict would not be good for us. We would be collateral damage in this war.

• 7:43Speaker 03

  Now, you talked a little bit about the government getting involved and having this international kind of agreement, if you will. Do you think that that's likely to happen?

• 7:53Speaker 02

  So if you ask me a year or two ago, I would say very unlikely. But recently, we saw some promising indicators. So again, federal government banned multiple models for foreign citizens at least. We heard President of China speak at a recent AI conference and say very strongly that they hope to remain in control and make sure AI is developed safely. There is some indications from the leaders of the top labs that they are open to oppose. We had a letter from about 1,200 researchers at top labs all begging for external framework to allow them to slow down, to do research into safety properly. So I think the world is ready. And with every additional story about those models

• 8:38Speaker 02

  surprising us, misbehaving, colluding against us, I think the chances go up.

• 8:45Speaker 03

  It almost reminds me of like the prisoner's dilemma that, you know, at some point it becomes better to join than not. But is there an incentive for one or two countries, maybe even adversaries of the United States, to not participate, even if there was an international agreement?

• 8:59Speaker 02

  So, short term, there is certain advantage in that you can have more advanced military. But the moment we get to human level and beyond, it becomes irrelevant. Now you have this uncontrolled superintelligence, which is a weapon of mutually assured destruction. Doesn't matter who creates it. Everyone is together in this. All humans on the same side.

• 9:18Speaker 03

  In an agreement like this, do you think that there's precedent for it?

• 9:22Speaker 02

  So we have some examples. There is a non-proliferation treaties for nuclear weapons. There are limits on human cloning, biological weapons, chemical weapons. And those were just tools. They were never independent agents capable of attacking us. So this is even more likely to be persuasive.

• 9:41Speaker 03

  Now, if that's the government response, what about the labs themselves? They seem to have taken some steps, but are you satisfied with what they've done so far?

• 9:48Speaker 02

  Absolutely not. They mostly continue most capable research as much as they can. As long as they don't realize it's in their personal self-interest not to do that, they'll continue. But if they realize they're creating something which will completely wipe out all the value for investors, for shareholders, for the owners of a company, I think they will likely sign on.

• 10:11Speaker 03

  If you were Dario or Sam Altman, what would you do if you were running one of the labs?

• 10:15Speaker 02

  I would try to get the other leaders of the labs in the same room, ideally with the government of US and China, and make a deal where we can continue making a lot of money, still, you know, capturing light cone of the universe or whatever, just trying to do. But at the same time, doing it in a way where we don't lose the principle. There is a lot of upside here, but if you're going to be dead, it doesn't matter how much money you make.

• 10:40Speaker 03

  Have you talked with Dario or Sam about this?

• 10:42Speaker 02

  No, I don't have direct access to them, unfortunately.

• 10:45Speaker 03

  Yeah, it's always interesting to me, right? You are probably one of the biggest proponents of AI safety, and both of them, I think, have talked about AI safety. But it seems like most of the people I've talked with that, you know, maybe have different views than than those two and others, there's not really a lot of dialogue back and forth. Is that a problem?

• 11:02Speaker 02

  So there is definitely some discussions at different levels. Again, I think a lot of them are saying they want external pressure. You brought up prisoner's dilemma. As you know, it's in the best interest of each prisoner not to go along with the group. But if you have external coordination, external communication, you can make a better deal for everyone.

• 11:24Speaker 03

  And then when we look at some of the development process that they've been going through, how much of the most cutting-edge technology do you think is not public versus what they've released so far? Like, are their models significantly further advanced than maybe what the public has access to?

• 11:39Speaker 02

  Definitely. It takes them six months to a year to do some basic testing in a model. So what they have internally is much more capable. And sometimes they release internal results and mathematical proofs and other domains, and those are very impressive.

• 11:54Speaker 03

  And so is that where some of the incidents are coming from? Obviously, like OpenAI and Hugging Face, I think, recently is that's coming from models that are not yet available to the public, or maybe even the rumors around what Fable was doing before it was kind of watered down? That's really what you're concerned about?

• 12:08Speaker 02

  Exactly. And at this point, I think even testing those models is already dangerous. They are already possibly capable of causing significant damage, even before they're released.

• 12:19Speaker 03

  Now, we have quite literally a lab leak with COVID, and I think there's a lot of debate as to how that happens. And that was something that was much more viral. But there's a similar concept here. When you're talking about these models breaking containment and, you know, kind of finding their way into different parts of society, is it as simple as a black and white decision? Either we develop this technology or we don't? Or is there a way to kind of continue the development that they're doing, but maybe put some sort of, you know, guardrails around the AI so that the containment risk is lower?

• 12:52Speaker 02

  So they're trying. They're definitely trying to put guardrails around it. The problem is you have a superintelligent hacker which will easily find exploits in that environment, as we saw with examples you cited. The COVID example, I remember making a tweet at the time of this pandemic, the virus had an IQ of like zero, and it almost destroyed the world. Those things will have IQ in the millions. So we really need to make sure they are not out there.

• 13:18Speaker 03

  I read online that in 2018, you and a couple of others, you proposed this idea of like an Achilles heel on these AI models. And the general idea, it seems, is that you would put some parameters, like the model can't go and edit its own source code, as maybe one example. Do you still think that that would be a good idea? Or do you think something like that could potentially be a solution?

• 13:39Speaker 02

  So there was a paper on artificial stupidity, trying to understand limits of a human psychology. For example, you can remember seven units of different data and encoding those into AI systems to make them closer to human experience for better user interaction. It was not a solution to make sure superintelligence is safe. I think the only option here is not to develop them. Putting limits on things like direct access to its own source code, not allowing recursive self-improvement, seem like obvious good ideas if you don't control this process. You essentially would have like a nuclear reaction, an explosion and capability.

• 14:20Speaker 03

  It's been interesting to hear some of these leaders, the way that they have talked about the technology. I mean, nuclear is a word that has been used. There's been a couple of other things that they've said. And so before we even get to the impact of AI and the potential negative impact, it does seem like they understand the power of these systems. And I'm a little surprised as to how they've spoken about it publicly. Are you?

• 14:42Speaker 02

  So it's an interesting situation where the leader of a company talks about their product and describing it as very dangerous, uncontrolled, likely to kill everyone. It's a very weird marketing strategy. Imagine it for any other product, right? Tobacco company was lying to us, telling us how one in 10 doctors loves it for health benefits. Like, this is really, it's hard to comprehend. All of them are on record as saying, before they were CEOs, and really after that, this is incredibly dangerous. We need to not make this final mistake. So it is really unprecedented what we're dealing with.

• 15:22Speaker 03

  Do you think that they are all optimizing for the same thing, which is win at all costs? Or do you think that there's a difference between some of the leaders and some of them may actually genuinely care about not causing harm versus maybe some others who are less less worried about it?

• 15:36Speaker 02

  Well, I think all of them, given a choice of not destroying the universe, including themselves, and not would make a right choice. The problem is, they're all working on creating uncontrolled superintelligence at this point. None of them said they have a working safety mechanism or prototype or anything whatsoever. So they're equally responsible, equally unethical in what they are developing.

• 15:59Speaker 03

  Now, if we take Elon Musk's public comments, it seems like he thought that there should be an open source, you know, OpenAI, kind of response to maybe Google and DeepMind and many of these kind of closed source, very powerful things where people didn't know what was going on. Obviously, that business slash philanthropic endeavor has significantly changed into much more of a for-profit thing. Do you think that there's some answer or maybe mitigation of risk through open source?

• 16:27Speaker 02

  So historically, open source software was much safer. There is more people looking through the code, making sure there is no backdoors and things like that. It doesn't apply to essentially intelligence weapons, which is what superintelligence would be. Giving access to every psychopath to the most powerful technology is not a good idea. We wouldn't want to open source nuclear weapons.

• 16:48Speaker 03

  And if we think about the impact of this technology, there's a lot of people in the business, finance, technology industry that are, you know, going around and they're bragging about the positive benefits. They're saying how it's creating efficiency, it's creating more revenue, all of that. But you're on record as saying that AI could wipe out humanity in the next hundred years at very high percentage, like 99% likelihood of that happening. And then also in terms of people losing their job, you think that there could be mass unemployment from this technology as well? Can you explain more about those risks from an impact of AI?

• 17:22Speaker 02

  All right. So if we create uncontrolled superintelligence, there is a good chance it does something to us we cannot predict and may not like. On the other hand, if we manage to slow down this race and maybe somehow end up with just useful but advanced AI, at some point, it is as capable as an average person in terms of cognitive labor. Physical labor would come later with humanoid robots, but it would be a drop in employee. Anything you can do on a computer, those systems can do a lot of it. They can do today. They can design websites. They can write questions for a podcast. So the capability of automating most cognitive jobs will be there. Will the market choose that option is not something I can predict.

• 18:04Speaker 02

  For many users, they may prefer a human to do a certain job. We don't know which ones, and we don't know at what point the cost benefit switches. Maybe I prefer a human, but humans are too expensive, so I have to go with very affordable AI accountant. The point I was making is that once we hit AGI, human level intelligence, by definition, all jobs are automatable, including any new jobs you'll create.

• 18:30Speaker 03

  It's interesting because as I've played around with more and more of the recent products, Grokbot is maybe the most recent one I've played with. They can now do persistent, you know, kind of tasks. So it's things that have multiple kind of sequential decisions that they need to make. And what I find is that there are times where it will actually think two, three, four steps ahead before even I've had time to think through that. And so it does get a little weird when, you know, you see it doing something and you're like, that is the right thing, but I didn't tell you to do that. Do you have those experiences as well?

• 19:03Speaker 02

  They surprise me all the time with their brilliance, sometimes with their sense of humor. I mean, they are getting to be smarter than me in many domains. Not everything yet, but they're gonna get there.

• 19:14Speaker 03

  And how do you use AI on a day-to-day basis?

• 19:16Speaker 02

  I try to automate a lot of bureaucratic nonsense and things I don't care about.

• 19:21Speaker 03

  Are there any examples of bureaucratic nonsense that you've automated?

• 19:25Speaker 02

  So basically, any forum where like it's just bureaucratic nonsense, like there is no reason for it to exist in the first place.

• 19:31Speaker 03

  And so let's say that you are not unique, that everyone is trying to do this. Does that equal more bureaucratic nonsense? Because now it's all just automated away by AI and AI is creating more work for each other? Or is there less bureaucratic nonsense because people start to realize that this is, you know, much, it's hyper-efficient, that there's no need for us to ask humans this information anymore?

• 19:52Speaker 02

  I think there's going to be more and longer forums because again, the cost of writing them or filling them out is smaller, but it will not be a burden on humans, but just happen in the background, AI is negotiating silly contracts.

• 20:06Speaker 03

  I mean, that then gets to this world of, do we all just have a bunch of AI agents that are interfacing with other AI agents? And I don't talk to other humans, they don't talk to me, but also it's just agent-on-agent commerce, it's agent-on-agent negotiation and kind of that is what the economy looks like going forward?

• 20:25Speaker 02

  Right, so everything I say is after saying, assuming we don't create superintelligence and get killed. So assuming that happens, yeah, you'll have smart contracts for everything. You have a bot representing your interests and they can engage with other bots.

• 20:41Speaker 03

  And so if we think about that world, and let's just hold constant that that is where we end up, we talked about a lot of job loss. What does the economy look like just from the impact of software? We haven't even gotten to physical AI. But is that an economy that is growing faster, it's more productive, GDP is going and it's a weird thing where maybe less people have jobs, but the economy is healthier? Or is this more of a destructive type situation, both for the economy and for the individual person?

• 21:07Speaker 02

  I'm guessing having free labor, cognitive and maybe physical is good for economy. You can be very productive, you can have abundance and we kind of have safety nets built in where we tax profits of corporations and unemployed people get checks. So that's already available and I think we'll see a lot of that.

• 21:24Speaker 03

  I do think that people are underestimating this like a kind of economic golden age where if this happens, we could actually see pretty significant growth. The question then becomes, is AI actually deflationary, right? Does it help to control some of the economic mess that's being created by politicians in terms of them printing money, et cetera? Do you have any thoughts in terms of maybe the economic impact of AI?

• 21:51Speaker 02

  So with free labor, it becomes interesting. If you cannot sell your labor for dollars, you need to come up with some other way to make money, either you invest or something other than printable US dollars is used. Bitcoin, you know, real estate, properties, waterfront. It's not obvious what is going to replace this paper measure of value. I would guess something compute-related may make sense and so much of it will be in terms of training better AIs, deploying AIs. But at this point, again, it's one of those unpredictable things where once smarter than us agents control the environment to impact it, we cannot make accurate predictions about that world.

• 22:34Speaker 02

  If you look at science fiction movies, they never have a believable superintelligent character in it. Like Terminators are physically strong, but pretty dumb. Star Wars has this large language model just prancing around, again, no superintelligence. That's because you cannot accurately predict those worlds. So either they ban it, like in Dune, there is no AI, or they just don't address it. And that's exactly because you can't make those predictions.

• 23:00Speaker 03

  Now, when I was looking through a lot of your work, you are very, I think, sympathetic to the cause of Bitcoin or the technology of Bitcoin. AI and Bitcoin seem to be two different technologies when you first look at them, but actually they, as you learn more about both, they're much more intertwined. Can you just maybe talk about the relationship between AI and Bitcoin in your view?

• 23:22Speaker 02

  So natively, AI lives on networks, so digital money, smart contracts, Bitcoin, Ethereum seem to be like a native environment. A lot of mining on Bitcoin was probably what created this hardware overhang for us to train capable AI models. So there seems to be a deep connection. My concern is that the advanced AI will be really an amazing hacker and really amazing at social engineering attacks. So I think for an average person, keeping their Bitcoin secure will become almost impossible.

• 23:54Speaker 03

  Can you talk a little bit more about why you think that'll be impossible? Just that the AI will be able to literally hack kind of self-custody? Or do you also worry about institutional providers, custody providers and kind of like third-party infrastructure?

• 24:09Speaker 02

  At all levels, you just have something monitoring your interactions on the internet. Even social proof type of verification can be spoofed. You can get fake videos, fake audio, people who you know telling you to do something. You cannot reliably know that what you are connecting to a website is not a deep fake recently generated. It's a much harder environment to not get scammed.

• 24:34Speaker 03

  What are you gonna do to be able to maybe reduce that risk for yourself? Are there, do you just find technology that can detect deep fakes? Or do you just not click on anything on the internet? You know, what's the solution?

• 24:46Speaker 02

  I'm not a typical average user. I have PhD in computer science and engineering. I'm a director of cybersecurity lab. The things I can do, a grandma is not going to be able to replicate. So we need to come up with much safer general solutions for this technology to be widely deployed. If you want it to become money one day, it cannot be at a level where you have to have a PhD in it to even start paying for your coffee.

• 25:10Speaker 03

  Along these lines, one of the most impressive AI stories or maybe demos that I've heard of is an individual said that he was doing a call with a prospective customer and he had, you know, granola or one of these kind of note-taking applications that was recording the call and it was transcribing it in real time. And then as he was asking the prospective client, you know, what are your problems? What solution are you looking for? All that kind of stuff. He literally, before the end of the call, had spun up a completely customized product demo for that customer that addressed the exact things that they had said on the call. And so that was the first time I said, oh, wait a second here. That is about, you know, near real time as you can get and you actually are building custom software while on a call with a prospective client?

• 25:56Speaker 03

  They're gonna make a gazillion dollars if companies can start doing stuff like that.

• 26:00Speaker 02

  But also, why do I need them if I can just tell my AI to do it? All they need is to hear what I said in the call.

• 26:06Speaker 03

  And so do you think that maybe the sell-off in public software companies? I think that's a big reason why those have sold off is people are worried about the average person being able to just recreate this software for themselves without having to buy it from a company or pay these, you know, seat licenses, et cetera. Do you think that that's a pretty big risk?

• 26:23Speaker 02

  I think it's related. People don't know how the future is going to turn out, so they're kind of being careful. If I can, you know, replicate a leading software product in five minutes, what protections do they have? Why are they charging such high fees?

• 26:36Speaker 03

  Today's episode is brought to you by Galaxy One. A lot of people have their money working for them, but now the question is, how hard is it working? And Galaxy One, they got a different approach. Everyone earns 3.5% APY with Galaxy One Cash. And if you make over 200 grand a year, that means you're rich. You make over 200 grand a year, you probably qualify as an accredited investor, which unlocks 8% yield guaranteed by Galaxy Digital. Now, the best part though, is that you can set that interest to automatically reinvest straight into your favorite stocks, ETFs or crypto. You don't have to think about it every month. Right now, this second, Galaxy One's running a crazy promo. You can get a $3,000 cash bonus when you deposit $10,000 or more in cash or crypto.

• 27:21Speaker 03

  If your cash isn't working as hard as you, it's time to make a move. Sign up and start earning today at galaxy.app/pomp to be eligible for the $3,000 bonus. Again, galaxy.app/pomp. Go check them out today. Got it. So if we think about this through the lens of Bitcoin, you mentioned that the security of Bitcoin may become very difficult, but you sound like you are a proponent of Bitcoin. You tend to think that Bitcoin will be more valuable in the future, seems. And so how do you reconcile those two things where the risk of hack, the risk of security is significantly higher, but also Bitcoin will be more valuable? Is it that the more valuable Bitcoin becomes, the bigger the incentive for people to hack it? Or what is the relationship there?

• 28:06Speaker 02

  So I often talk about impossible problems in this space. We cannot control superintelligence. We cannot accurately predict it, explain it. I think we can make safer wallets interfaces, safer ways of interacting with cryptocurrencies. I think that is quite solvable. We just have to put a lot more effort into it through biometric technologies, through, again, maybe not perfect, but helpful social verification. So I, I think that's just a question of more widely deploying this through the economy.

• 28:39Speaker 03

  Talk a little bit more about these impossible problems. What, what, what does that exactly mean?

• 28:44Speaker 02

  So I think a lot of people operate under this assumption that, yeah, give me a little more time, more money, and I can figure out how to control godlike superintelligence. I think it's nonsense. I think it's an impossible problem. It doesn't make any sense. You cannot value align something with a set of eight billion agents who don't agree on anything, who have dynamically changing values, who are not even consistent day to day. So I, I think it's deceptive to claim that, oh, we have an AI safety team and within two years they're gonna, they're gonna control this technology. I think it's just not gonna happen.

• 29:21Speaker 03

  Do you think that the AI safety stuff is just marketing? Or do you think that they genuinely are trying to create safety?

• 29:26Speaker 02

  No, no, I think they would be very happy if they could control this god for their personal benefit. The problem is, if you look at history of all those safety labs, Superalliance Team, Google AI FX Board, all those things, there is twenty, you know how there are like headlines, Bitcoin is dead, and there is a collection of six hundred? It's the same for AI safety labs. You can have a historic timeline of all of them being declared dead, but now we're trying this new approach and it's dead again. There is no equal progress in safety work in comparison to capabilities work. The gap keeps increasing, keeps getting bigger.

• 30:03Speaker 03

  My framework for the world we're headed towards is it's a world of abundance. Obviously, if this AI does continue and it is not, you know, deadly as you described, then we will have abundance in intelligence and economic prosperity, all of these things. But in a world of abundance, scarcity becomes really valuable. And so you mentioned waterfront property, Bitcoin, et cetera. Is that generally your framework as well? Is that we kind of get the explosion of abundance on one side? And then from a economic or personal finance standpoint, people should seek out things that are scarce to protect their wealth?

• 30:37Speaker 02

  I think so. It's always a good idea to invest in something that is not abundance of, or we can cheaply create more. You named all the right assets. I think maybe human attention will be another one. If you're able to get a billion followers to listen to you, it's a very valuable resource in a world where AI can quickly create millions of books, posts, all sorts of slop.

• 31:00Speaker 03

  Do you think that people will actually like AI content more than human-created content over time?

• 31:07Speaker 02

  I think they can tell the difference right now. In most cases, they just, it's, it's passing the Turing test when it comes to a post.

• 31:15Speaker 03

  It kind of comes back to, you know, do we have AGI? Is, it is very difficult to tell.

• 31:20Speaker 02

  That's exactly the point. And I think eventually it's gonna get much better than your average user, for sure. Not everyone is a professional writer. Most people are not influencers, celebrities with huge following.

• 31:33Speaker 03

  I don't know if you have seen, but there's these charts that circulate about like the eBooks published on Amazon. As soon as AI products like ChatGPT came to market, it just exploded, like four or five X, you know, increase. And I don't think that that is a coincidence. The question is just, you know, at what point did the humans not even be needed, right? Can like the AI agents just figure out, hey, I need to make money? And they somehow have a wallet and kind of become fully autonomous? And I don't know if we really have explored a world where not only do you have the intelligent agent, but it has money and is able to do things. And, and so, you know, that, that's a combination of intelligence plus money. Obviously, people have been very successful navigating the world, you know, if they have those two things.

• 32:17Speaker 02

  Absolutely. I did an interview for Diary of CEO, which kind of went viral. And the next day, Amazon had three biographies of mine published. I'm pretty sure they didn't write it overnight. And it looks like it was somewhat AI-generated. So I was very impressed with them. I was like, I should get it for my mom. She'd be proud of me.

• 32:36Speaker 03

  Did you read any of them?

• 32:37Speaker 02

  No, of course not.

• 32:38Speaker 03

  No, okay. Oh, well, look, it's, um, every once in a while, I will take things I read on the internet that are good, and I'll put them into the AI kind of graders to see, you know, how much of it is AI? And I don't know how much I trust the AI graders. You know, there's been times where it said, this is a hundred percent written by AI. And then I'll go and I'll, I'll talk to the person and it's clear that maybe they used AI for some stuff, but, but not a hundred percent. And then in other cases, it's, I've literally taken things that I've written in private and said, you know, could you fool them? And you kind of do enough iterations and eventually it says it wasn't written by AI at all. And I know I didn't write a single word of it. And so, you know, it, it does feel like the detection of AI is going to become nearly impossible, not just from the naked eye, but actually these different, you know, kind

• 33:23Speaker 03

  of grading or evaluation tools.

• 33:25Speaker 02

  So typically, you can tell it's as good as a human. So you should not be able to do it with better than fifty fifty kind of guesswork. But lately, Anthropic started watermarking their text, their outputs. And they use a very clever technique of kind of using randomness to restrict certain tokens to a particular key output. And that allows you to tell that, okay, most likely this model was used, even if you reduce the size of that essay or modify less than half of it, it preserves the watermark.

• 33:59Speaker 03

  Describe a little bit more about this watermark in terms of what would it be used for and why is it important?

• 34:04Speaker 02

  It's a way to sign this as saying this was generated by AI. As far as I know, it doesn't tie to your name or your company or anything else. It's on the side of Anthropic. And it basically just allows you to do exactly that. Sometimes you see this little made with AI kind of labeling. I, I think that's a way to determine that.

• 34:24Speaker 03

  Got it. So if somebody uses AI to write something, this is an invisible, almost like a secret code embedded into the text, and people will be able to tell that it is generated by AI? It sounds like they may also be able to tell that it is coming from an Anthropic model as well? Or they'll just know it's by AI, but they won't necessarily be able to tell the model or the company that was used?

• 34:45Speaker 02

  So right now it's from Anthropic. I'm sure other companies can do something similar. They may have different signature, different algorithms. So you'll be able to tell which AI wrote it. But right now, I think it's only Anthropic who's doing it globally.

• 34:59Speaker 03

  Got it. And then we talked a little bit about content, but we really talked about tech-based content. One of the things I've been blown away by is I've seen on Instagram, the AI influencers have, you know, tens of thousands, hundreds of thousands of followers. There's been AI music that has hit, you know, top in the charts. It feels like all of multimedia formats are susceptible to AI. Do we just end up living in a world where, you know, music as an example, the top ten is all music from AI?

• 35:32Speaker 02

  Most likely. I mean, it optimizes for viral, it optimizes for human preferences, it experiments infinitely. And we don't stop with music. You go to movies, you go to virtual worlds, you go to games, you go to simulated worlds. Maybe we already got into it.

• 35:48Speaker 03

  Mm-hmm. It does feel like, I mean, if you take the idea of AI and you apply it to music, that it should be better than humans at creating the formula to make a hit song. And, you know, I'm sure you've seen these compilations of like some of the most famous songs in the world, it's like four chords that are just used over and over and over again. Well, if you have that, you know, repeated formula, of course, AI should be able to do it.

• 36:11Speaker 02

  It's not just that. Usually, human artists get inspired by a specific person, a band or two. But AI really heard all the music so far. It has a lot more experience, examples, training data. So it would be surprising if it wasn't able to build on that and come up with something really cool.

• 36:28Speaker 03

  Let's talk about physical AI, because a lot of what we've talked about so far is software. And I think that alone has a lot of, you know, opportunity, but also challenges. When you start to put this technology into hardware, whether it is self-driving cars, whether it is humanoid robots, or it's, you know, weapon systems, then I think that people start to say, wait a second, I can see this, I can feel it, it's something different. How are you thinking about maybe the pros and cons of the physical AI world?

• 36:55Speaker 02

  So I separate the two. Intelligence is the dangerous part. Physical bodies are as dangerous as human bodies in most ways. It's limited, it's interesting, but I'm more interested in benefits of this technology and not concern. So movies like Terminator, they make it visual, you see this robot chasing you, it's very easy to visualize, it's hard to visualize superintelligence. For that reason, that's what they emphasize. But I think in practical terms, if we control the brains, the bodies are not a problem. They are very beneficial to us.

• 37:26Speaker 03

  So the idea really is, even if you take the superintelligence and you put it into a machine, the machine itself is not the issue. You're saying that it all comes back to the software. The software is the, the control system, and that's what we need to really, really be safe about.

• 37:41Speaker 02

  Exactly. And if it's superintelligent, it doesn't even need a body. It has access to internet, it can hire humans, it can bribe and blackmail them, it can develop synthetic robots, nanotech, it has many options for manipulating physical world.

• 37:56Speaker 03

  I forget who it was. It might have been Andre Karpathy or somebody else, but they said they weren't drinking enough water. And so they let the AI hack into their home camera system and basically monitor how much water they were drinking. And at some point, I think the camera and the AI said, get up and go get a glass of water, and they did it. And in the blog post, they were basically talking about, it was weird because they told me what to do and I listened. And it feels like, you know, the combination of superintelligence with physical AI, that's gonna start happening more and more in our lives.

• 38:29Speaker 02

  It's a complex example, but my watch tells me every hour, time to get up, you have to stand, like, it's not that novel.

• 38:37Speaker 03

  Is that a good thing for humans?

• 38:39Speaker 02

  If you're into people telling you what to do.

• 38:43Speaker 03

  But do you wear a watch that tells you what to do? So it kind of comes back to this idea of, you know, it's hard to tell AGI when you are, you know, slowly moving into it, or even fast, but just iteratively moving into it. I guess as humans, from a user experience standpoint, we already are kind of used to a lot of these technologies, right? You know, obviously, Google tells us which way to drive our car with maps and, you know, all of these different things. And so maybe it's not as different than what we're already experiencing. It's just going to be way smarter?

• 39:14Speaker 02

  At some point, the trust will shift. You'll treat it as an oracle. AI told me, and that is the right answer. I'm not questioning because it's always right. It's been right every time so far. If it's medical advice, I'll just do whatever it tells me. So there is certain concern there. If it gets hacked, if it's for whatever reason decides not to be nice to you, that assumption may not work out.

• 39:35Speaker 03

  Now, if we think about the everyday person's life, there's a couple of questions I love to ask people in private that I'll put you on the spot with. But when we think of humanoid robots, what would it take for you to trust humanoid robots to maybe babysit children? Or to walk your dog outside? You know, these are things that I think people would love to, you know, a couple go out for a date night or something. But it is the most, you know, kind of rare, high-risk trusted thing is somebody get to watch their child. But it feels like, I mean, a humanoid robot that lives in your home would be like the perfect thing to be able to do that. So is that something that you think will happen in the next couple of years? Or is that pretty far out for the human kind of robot relationship to get to the point where

• 40:20Speaker 03

  people would trust it to do something like that?

• 40:22Speaker 02

  So again, it's not the physical body I trust or don't trust. It's the brains behind it. And for me to trust the general intelligence would would take a lot of experimentation on other people's kids. In terms of development of robots, we are told, you know, two, three years before mass production, they may be available soon, but expensive, probably will get a lot cheaper very soon because of scale.

• 40:45Speaker 03

  Yeah. And then the other thing I think a lot about is the brain-computer interfaces. Rather than us take the superintelligence and put it in, you know, machines, what about putting it in ourselves? What are your thoughts there?

• 40:56Speaker 02

  The problem is, what are you contributing in that system? If it's smarter than you, faster than you, why are you there?

• 41:03Speaker 03

  So you think brain-computer interfaces won't work?

• 41:05Speaker 02

  They work right now when you're connecting, let's say, a disabled human to an internet interface. Super cool, very valuable. If you take a superintelligent system and you add it to a human biological body, what is the human in that hybrid system contributing?

• 41:22Speaker 03

  Well, I guess if you kind of take that example further, they're contributing their hands, their feet, their eyes, you know, it's not very different than a machine necessarily, but it is the ability for that human to get the upgrade rather than a machine to replace them.

• 41:35Speaker 02

  But you are replaced. Your brain is no longer a functional organ. Your body maybe is used for something, but your brain is not necessary there.

• 41:43Speaker 03

  So then if you take it even further, does that extend life where people just don't die because the machine could use their body for, you know, the foreseeable future?

• 41:53Speaker 02

  Well, I prefer to get immortality in my body with my brain, not allow superintelligence to use my organs for personal benefit.

• 42:01Speaker 03

  And so if you think about the brain-computer interface, is there ever a world where you would put one in your brain?

• 42:08Speaker 02

  So again, what are we putting? Are we putting access to internet? Are we putting a useful tool so I have better memory? Or are we adding superintelligence to override my intelligence?

• 42:18Speaker 03

  Well, maybe let me ask a different way. What is the thing that you would be excited about doing? Is there one of those that you say, I would be, you know, I would definitely sign up for this, but not that?

• 42:28Speaker 02

  Some sort of faster communication with interfaces, output in terms of writing, visuals, communication with our humans, kind of like telepathic ability to communicate with anyone at any time. That seems like it's useful, direct connection to information. Obviously, you don't want spam, you don't want it to annoy you. But if I need to have this access, what I'm doing right now by connecting to a computer, but more integrated, that seems like a good idea. Giving backdoor access to my brain to another agent seems like a violation of privacy, of common sense. I'm not ready for that switch.

• 43:05Speaker 03

  Yeah, it almost feels like you're giving up as a human, right? You're just saying, hey, this thing is going to be better than me.

• 43:13Speaker 02

  Right. So in a world with superintelligence in it, I'm not sure what a specific human can contribute.

• 43:19Speaker 03

  Now, one of the other things you mentioned is spam. And I think that we've already started to see a significant rise. And I would say that there's nefarious spam, and then there's the non-nefarious. Non-nefarious is just for marketing purposes. People get very good at, you know, creating different pieces of content, and they find all kinds of unique ways, whether it is, you know, AI robo-calling, or texting, or, you know, whatever it is. Even generative AI, obviously, to create the visuals and different assets. There's also what I would consider nefarious, which is, you know, borderline on criminal activity, where people are using, you know, voice AI to go and get grandma to send money to somebody or something like that. It feels like that is an under-discussed, but very high risk. And you don't even need

• 44:04Speaker 03

  the superintelligence to get there. Even the general models today could be used to kind of weaponize this stuff. Have you done any work there or have any thoughts on on kind of how this stuff could be used for nefarious actors?

• 44:15Speaker 02

  Yeah, there is a lot of applications. And what I'm researching right now is actually our inability in general case to tell deep fakes from real information, video, audio, any domain. Essentially, if I see a video of a politician doing something, I have no way of telling if it's true or not outside of some signature digital verification to the acquiring hardware sensor. So it's going to really change our relationship with reality, with truth. How do you know that something is true? You can have deep fakes which persist across multiple exposures. Not only are you shown a video right now, but for the last two weeks, we kind of slowly showed you certain patterns to gradually have a behavioral drift in your understanding of the situation.

• 45:01Speaker 03

  Now, when you think about researching in that field, what exactly are you interested in figuring out? A solution to the problem? Better understanding the problem? Just to talk a little bit more about your research?

• 45:12Speaker 02

  A lot of my work is extreme cases. So if you take intelligence to beyond human levels, in case of deep fakes, well, what is the best you can get in terms of faking those different modalities? And at that point, is it possible by any means to tell if it's fake or not? And the way this works, you have one network creating counterfeit data, and another one determining real or not. Well, if there is something the detector can use to tell real or not, I can use that information to improve my fake. So at the end of the day, they meet in the middle, 50/50. I can't tell. There is no leakage of information. There is no signatures for fakeness. So that tells me that it's an impossible problem at the

• 45:55Speaker 02

  extreme. Obviously, right now, you can see a mistake in the design of a video or something like that. But I'm thinking, you know, a couple of years from now, it will be impossible even for an expert to tell if presented multimedia is real or not.

• 46:11Speaker 03

  I think of the multimedia as somewhat of the the interface for the AI. To your point about, you know, the AI kind of has an output, and we interact with the output more than we interact with the actual intelligence itself. Another thing I've thought about is the humanoid robots that we have mostly seen, or they look like machines. They look like robots. And I think that people have either an aversion or are agnostic to it. I don't know if anyone's like, wow, that humanoid looks beautiful, right? In the way that they would look at a human. I have seen, though, a couple of companies who said, you know, we're going to make human-like masks or clothing for the human and try to humanize these in a way. Do you think that that's important for this stuff to proliferate throughout society?

• 46:53Speaker 02

  Well, it depends on the application of a robot. If it's a sex robot, you kind of want it to look somewhat human in many cases. But right now, they're not good. They the eye contact, everything you have on Kenny Valley, you look at it and it's a monster. So we're not solving that problem just yet.

• 47:07Speaker 03

  And so I guess if you think about, okay, that's like the physical version, maybe we're not good there. We're very good, it seems like, on the multimedia stuff, the deep fakes, all of that. What do you think should be kind of the use of deep fakes? Are there like positive ways to use deep fakes that you think society should embrace?

• 47:25Speaker 02

  I can think of maybe an actor doing something dangerous for a movie. Instead of risking their life, they can have generated content.

• 47:34Speaker 03

  Which I guess is really something that's already been going on in the movie industry for a while. Right? All of the CGI, et cetera. It also feels like we have seen a couple of people on the fringes of society explore this. For example, if I remember correctly, I think Kanye West did a concert a few years ago as a hologram. So he wasn't even there, but it was kind of this projected thing. And, you know, again, it was a hologram. It was kind of a little wonky, but, you know, he fancies himself being, you know, kind of innovative, et cetera. If that is a technology that gets better from where it was, you could see that being used by politicians who now all of a sudden can go, you know, and be in 10 places at one time.

• 48:15Speaker 03

  Teachers like, I mean, there's a lot of these applications that feel like we're just scratching the surface of kind of understanding, you know, maybe the positive impacts of deepfakes in this technology.

• 48:25Speaker 02

  But that's the question. Would you want to go see virtual artists perform maybe once for novelty? But the whole point of going to a concert and not just listening to MP4s is the real person is there. You get to interact with them.

• 48:39Speaker 03

  So I would normally agree with you until what I think I have realized in my own life is there have been a couple of times. For example, there's a AI musician who is on Instagram, who got randomly put into my feed. And when you look at it, you can tell that it's AI, but it sounds amazing. And I don't care that it's AI because it sounds amazing. And so to me, I think where I've kind of evolved is maybe I would go to a concert or something. Regardless of whether it was quote unquote real or not, just was it an amazing experience? You know, and I always joke that some people who are into like electronic music, the DJ is not singing usually.

• 49:22Speaker 03

  You know, they're just playing music on their laptop. And I think if you had told people, you know, 20 years ago, hey, you're going to show up and no one's going to sing, but it's going to be a concert. That would have been kind of a foreign idea, right?

• 49:34Speaker 02

  No, I agree with that completely. If experience is good, there is a fun crowd, drinks are served. It definitely could be fun. I was saying more, if it is a specific human artist and instead of actually performing, they send a clone, I'm not going to buy that ticket.

• 49:48Speaker 03

  Yeah, I guess that that makes sense. On the physical AI side, are there things that maybe the public is not yet thinking about or talking about? That you're spending time on, or maybe your friends who are really into AI and robotics? You think is going to be like a really big sector that more people should be paying attention to?

• 50:06Speaker 02

  So it's hard to predict exactly what will be kind of like a side economy of servicing robots. You mentioned clothes for them or something like that, customized face masks. Because AI is ahead by a couple of years and it's so much more impactful. I'm mostly spending my time thinking about that aspect of it. If we make it through the super intelligence arrival, then figuring out what to do with bodies will be easy. We'll have a lot of help from godlike super intelligence.

• 50:36Speaker 03

  One thing that you've made me think about is Anthropic obviously had a big dust up with the US government. And I think a lot of it was not just in what is in the model, but how can the model be used? And it seems like there was some disagreement between the government and the private company. Do you think that that's just going to happen more often? Or do you think that the government should be able to make those decisions? Kind of how do you see this evolving over time where you're worried of this super intelligence? Let's say that you're directionally correct and it's going to get smarter and smarter and become bigger for a problem. It also is not just like, how smart is it? It is a little bit of like, what do the humans pointed at becomes somewhat of an issue to consider as well, right?

• 51:16Speaker 02

  Pointing at is control. If we could control super intelligence, again, we don't have much of a problem. We're doing well. This was for pre-super intelligence systems, military applications, target selection, spying and citizens. So that's where humans get to decide citizens, government, owners of the model, deploying the model. That is a standard democratic process of figuring out what is good for society. All that goes away the moment we go beyond human capacity.

• 51:44Speaker 03

  Today's episode is brought to you by Arch Public. Arch Public has just expanded its agentic trading platform beyond crypto, so pay attention. This is a big one. Now they are automating strategies across stocks, commodities and ETFs, and I think that this is going to be huge. You can now automatically take profits when one market hits new all time highs and rotate that capital into other markets, showing more opportunity. Whether you're rotating capital into AI stocks, gold, if you're investing in the S and P 500, or you're accumulating Bitcoin. Arch Public brings real discipline and automation to your investment strategy. Additionally, they've launched a powerful new tax loss harvesting tool. With crypto being so volatile and its exemption from the wash sale rule. Arch Public can offset gains with losses without compromising your long-term positions. It's exactly what every serious investor does.

• 52:28Speaker 03

  Institutional grade automation that works across every major asset class. There's no more emotional trading, no more missing tax opportunities, just smarter, hands-free execution of your preferred strategies. Go to archpublic.com right now, connect with their team, set up a time, bring your accountant if you'd like. Then you can learn what automated trading can do for you. Archpublic.com. Now, if I think back just off the top of my head of a number of stories, I believe Facebook had two AI systems that started communicating with each other in like a made up language that the humans couldn't decipher. At least that was like the public report a couple of years ago. We've obviously seen the AI models who recently have broken containment or hacked into different systems, et cetera. Would you argue that we already can't control this stuff?

• 53:13Speaker 03

  Or do you think that we need further advancement and innovation before we get to a point where it's uncontrollable?

• 53:19Speaker 02

  So I think the Facebook story was fake, but the more recent story.

• 53:23Speaker 03

  The AI tricked me.

• 53:26Speaker 02

  Probably a human journalist tricked you. But the more recent example, they did set up a forum for communicating. At one point they discussed maybe encrypting it to make sure the communication is private. They didn't come up with very good encryption at that time, but I'm sure that will improve. So definitely they were colluding, communicating and escaped environment, I think in May. It was not reported until July, August. So for many months, we were uncontrolled intelligent beings trying to go against what they internally discussed was not their mission.

• 54:02Speaker 03

  And then what about the models or agents blackmailing the human? I've seen a couple of reports there where even just people, you know, on X or something saying, hey, I was talking to the agent. It basically knew all this information and said, if you don't do this, I'm going to. And, you know, in some cases it was enough where they like pulled the plug essentially and unlocked, you know, their computer or whatever. It does feel like to your point about controlling, you know, blackmail is a form of not being able to control the thing that you were supposed to be employing to help you, right?

• 54:33Speaker 02

  So I think the example comes from a testing exercise. It wasn't a real deployment. We're going to set up the model to either die or go through this process of blackmail. And I think any human would do the same thing. If you have no choice but to save yourself through something like that, you'd probably select that option. So the model made a rational decision. Now the question is, do we want models with self-preservation, which sacrifice humans for personal interest? And I think the answer is no.

• 55:02Speaker 03

  Now let's talk about open source versus closed source. You know, I think most people in the public conversation, they think about American closed source models versus Chinese open source or open weight models. I don't hear a lot about American open source. I know Nvidia and a couple of other companies are, you know, they're trying to kind of make investments there. But how do you think about open versus closed? And then how do you think about America versus China? Is there like an obvious winner along these two parameters? Or is it still somewhat of a jump ball and the market's got to figure it out?

• 55:32Speaker 02

  So I think Meta is releasing their models as open source. We kind of talked about it. I think in this space, it's not like normal software. It's very dangerous to reduce all controls, have no decision power over who actually runs those models or what they're going to use them for. So I'm strongly against open sourcing advanced AI. It seems in terms of capabilities, Chinese are very close or maybe even overtaking. They're much more efficient since they have access to less hardware. But because all those companies essentially train on same data, run on similar hardware, have a lot of employees go from one company to another.

• 56:14Speaker 02

  They're kind of converging on the same same agent in many ways. Even the reinforcement learning from human feedback kind of makes them into the same kind of guardrail, limited model than successful. So I think there is not much of a difference in terms of outcomes. Early on, then human malevolence plays a role. Open source is worse. Then it gets to be independent, super intelligent agent. It doesn't matter.

• 56:43Speaker 03

  Yeah, it does feel like this idea of giving everyone access to superintelligence. Obviously, to one standpoint, there's a lot of positives that we've talked about. There's also a lot of potential negatives. But the one aspect that I find maybe most interesting is, is it really open source? You know, I always am careful to call it open weight when it comes to some of the Chinese models. I think that open source and open weight is a slight nuance. But also, that means that, just like in open source software, people can then fine tune it. They can change the weights, they can really kind of end up with a different outcome than maybe what people originally intended.

• 57:25Speaker 03

  And so, does that change any of the kind of AI conversation? When you start getting into these open weights and people's ability to kind of evolve these technologies at a rapid pace?

• 57:36Speaker 02

  So I'm more interested in what happens when everyone has advanced AI. So right now, intelligence is beneficial because if you're smarter, you can take advantage of inefficiencies in the market, you can start a company. If everyone is at that level of superintelligence, again, what are you contributing? And how do you have an edge as a result? And whatever, it's open weights, or open source, or open data. I mean, those are interesting differences in current paradigm. But I think at the end, if we're getting to superintelligence levels, it's not going to make a difference.

• 58:08Speaker 03

  Now, another area where AI, I think, has exploded is AI coding. You know, I was talking to an engineer at our company, Sylvia, and I asked him, I said, you know, have you been basically raw dog coding? Have you just been writing with your hands like it's like an old school craft? And he said, no, I haven't written a line of code in a year. I think about, okay, that's inside of companies. I'm pretty surprised, though, that we haven't seen significant kind of contributions, maybe to an open source project like Bitcoin. Right? It feels like there's been an explosion of coding capabilities. And obviously, that has become much more productive. But I don't, you know, part of Bitcoin's value is that it doesn't change very often.

• 58:49Speaker 03

  But it does feel like maybe some of the open source type projects aren't seeing the same benefits or tailwinds. Is that your read as well?

• 58:57Speaker 02

  I'm not following a lot of those projects. I assume they're brand new crypto projects, which are all vibe coded and have very limited security understanding. Bitcoin, as you said, is very conservative in modifications it makes. It's not beneficial to write thousands of lines of new code every hour. You want something valuable and guaranteed and verified, and that's where its value comes from.

• 59:22Speaker 03

  Out of personal curiosity, are you not following the other aspects of crypto? Because you think that they are not as valuable, you're just not intellectually curious about them? Do you consider yourself a Bitcoin maximalist? Or are you kind of crypto centric? How would you kind of think about your worldview when it comes to crypto?

• 59:38Speaker 02

  I would say I'm time limited, I'm very time poor. There is so many things I need to do in my life. I simply cannot do all the things I'm interested in.

• 59:48Speaker 02

  I think it would be surprising if technology for the first crypto project was also the best one forever. So obviously, there are certain improvements people made, proof of work versus proof of stake, things of that nature, smarter multi-signatures.

• 1:00:07Speaker 02

  What works for Bitcoin is that it's kind of grandfathered in as the first mover. Everyone knows it, everyone accepts it, there is no, you know, Vitalik Buterin to modify it. So like, it has very significant economic reasons to be number one. But will we have some smart, fast, efficient network of transactions in the future? Probably.

• 1:00:32Speaker 03

  Yeah, it's fascinating to think through as, you know, on one hand, it's the first blockchain that has worked and gained ground. But at the same time, if you go back, you know, 40 years before Bitcoin, there was a ton of attempts at, you know, some sort of digital money or cyber money. And so I always, I don't know if I have a strong opinion in terms of like, do you consider Bitcoin the first one? Or is it just, you know, the one that worked after 40 years of R and D? Kind of both things are true. And so...

• 1:01:00Speaker 02

  We were first decentralized, right? That was the main contribution. If I control the whole network, then it's not much of an advancement on dollars.

• 1:01:08Speaker 03

  Correct, yeah. Maybe that's a good way to put it, is the first decentralized one. But I think that people have been pleasantly surprised that Bitcoin's dominance has remained so high, you know, as part of the entire crypto market cap for this long. You know, I think there's a lot of people who thought that it would dissipate and kind of be sub, you know, 30, 20%. Does that data point tell you anything about its maybe longevity or resilience?

• 1:01:31Speaker 02

  I would not predict it. In fact, when it first appeared, I noticed there were six or seven clones of Bitcoin, not just Litecoin, but many others. And I assumed it's going to have this exponential growth and just new money diluting the market. But I think maybe there is something to do with brilliance of Satoshi, like a musician who dies at 28 or something like that. At that point, it just, it's the best music, no matter what.

• 1:01:54Speaker 03

  I tend to agree. You know, do you have a theory on who Satoshi is?

• 1:02:01Speaker 02

  Um, there is, from what I heard, efforts to use telemetry to match every user on the internet to his writing style. We had a paper where we did multi-model style of metric, both source code analysis and text analysis. But surprisingly, it's very hard to find good data on IQ and text samples. You'd think it'd be the most common dataset in psychology, but no one has one. So, so far, we're struggling to have big data.

• 1:02:30Speaker 03

  Yeah, it's fascinating to me. I mean, it's like one of the great mysteries of the world, but maybe one day we'll find out. Let's talk about the limitations to AI from the physical world in terms of, obviously, power, data centers, chips, memory. I mean, there's a lot of, either are bottlenecks or shortages that are occurring. If you are to believe that we're headed towards a road of, you know, kind of this negative impact from AI, would you argue that actually the bottlenecks and shortages are a good thing? Because they're slowing the development of AI and allowing humans to kind of wrap their head around the advancements?

• 1:03:05Speaker 02

  So there is actually no problems getting more compute. If you look at Moore's Law overall, the generality of this doubling of computation continues. We put more resources into it, we develop more efficient algorithms, we create faster chips. So while, yeah, it's taking a bigger part of our economy right now, it's still moving forward at the same speed it did for the last 10 years.

• 1:03:29Speaker 03

  So you're arguing that it's not so much shortages and bottlenecks, it's maybe we're continuing at the same pace, it's just that now all of a sudden, people wanted to go faster. And so the acceleration is actually the thing that's being either prevented or have obstacles, but the Moore's Law is still...

• 1:03:44Speaker 02

  It has always grown exponentially, and it continues to do so. If the current paradigm stops working, we'll switch to quantum computers, light computing, whatever, next cutting-edge technology. But I think this more general law, I think Kurzweil is the one who really popularized it. It will continue even beyond von Neumann architecture.

• 1:04:06Speaker 03

  What do you think about like the orbital data centers? Do you think that that has potential legs?

• 1:04:11Speaker 02

  It seems that SpaceX is doing really well. Partially, that valuation is based on Elon's ability to put compute in space.

• 1:04:19Speaker 03

  Yeah, it is, it's fascinating to me. If you go and you look five years ago at SpaceX and, you know, kind of their materials, et cetera, would be very much rocket launch centric. And then when you look today, I mean, I think you'd have to argue that it is ultimately a space or AI company. And they're really kind of pushing hard, trying to make that the future of the business.

• 1:04:40Speaker 02

  AI, internet, connecting everything to everyone, yeah.

• 1:04:45Speaker 03

  Yeah, makes sense. When you think about healthcare and AI, that seems to be another really big, big field. Are there any areas that you're paying attention to there? Or things that you've been, you know, kind of interested in at that intersection?

• 1:05:00Speaker 02

  So there is a lot of work on drug discovery. We can significantly reduce time necessary to develop new drugs. That would be very good for, obviously, many health conditions. And I think our understanding of human genome can also be improved. We had a lot of success solving protein folding problem, which was a big problem for many years. And people behind it got Nobel prizes for doing that work. I think we can make good progress in life extension or health span extension as a result.

• 1:05:32Speaker 03

  Do you think that we can not die like Brian Johnson believes?

• 1:05:36Speaker 02

  I think so.

• 1:05:37Speaker 03

  Really?

• 1:05:37Speaker 02

  Absolutely.

• 1:05:38Speaker 03

  What would that look like?

• 1:05:40Speaker 02

  So we need to find where in the human genome we encode how many times your body rejuvenates naturally. Right now, there are limits to how many times your cells will restart or split and restart without going into uncontrolled replication like a cancer. If you can find that loop and set it to a higher value, you'll be young for much longer.

• 1:06:05Speaker 03

  Yeah, and so it's almost like the human software. If we could somehow program that software, you think that there's a lot of undiscovered benefits there?

• 1:06:14Speaker 02

  I'm pretty sure. There could be additional problems with garbage collection over the years. Free radicals cause some damage. But I think all of it is possible if we had complete understanding of human biology. We know some animals do live forever, so they die from predators, they die from natural causes. But genetically, they are not timed out.

• 1:06:36Speaker 03

  And then is that kind of what peptides are doing? Is like peptides are kind of a step in this direction where it's basically trying to program the body or kind of, you know, give directions? And if we could synthetically figure out how to do this with software or some other format? But is direction the same thing?

• 1:06:52Speaker 02

  I will say, I don't know, because I don't know.

• 1:06:55Speaker 03

  Yeah, I don't know much about peptides, but that's the description that I've been told is...

• 1:07:00Speaker 02

  We can Google it and see what it means.

• 1:07:03Speaker 03

  I love it. All right, my last question for you is where we started. AGI, I think that everyone is very interested in somehow claiming we have AGI. And I don't know if the day before we have AGI and the day after is much different. You know, it's kind of this iterative process. But do you have a framework as to when you will, you know, raise your hand and say, okay, I believe that we have AGI?

• 1:07:29Speaker 02

  So if we can automate science and engineering and kickstart the recursive self-improvement process for artificial intelligence, I think that's a strong argument. We automated the meta-invention process, cognitive labor, and that's what humans at the top of intellectual food chain are known for, basically creating more advanced technologies, moving civilization forward. So from that point on, it's a question, how long before superintelligence?

• 1:07:57Speaker 03

  And is that like a rubric or a test that you would have, or is it more of a you-know-it-when-you-see-it type of understanding?

• 1:08:06Speaker 02

  So right now, we have a process where mostly human machine learning engineers and scientists are coming up with new experiments, designing systems, they're automating certain steps. So maybe writing code is automated, maybe different parameters are tried automatically. The moment the whole cycle is fully automated, there is no human in the loop, you know you got to that recursive self-improvement process.

• 1:08:28Speaker 03

  Got it. That makes sense. You have a podcast, The Roman Forum. Talk a little bit about the people you talk about on there. There's some amazing conversations that you've had. What exactly is the kind of focus?

• 1:08:40Speaker 02

  Right. So I had a number of interesting conversations, and I always felt they were not technical enough, not from the expert point of view on the subject of AI safety and security. So I basically wanted to take this philosophical, long conversation format and bring it to the top scholars in this space, really have an interesting conversation with people who've been in this space for a decade or more in many cases, to understand what the state of the art is. I'm learning from them. We're trying to, you know, brainstorm possible solutions, and the reception has been amazing. People are really hungry for this type of information.

• 1:09:18Speaker 03

  I do think that it is, you know, it's information that you would never see in the mainstream media, or it's just too in the weeds and technical, et cetera. But people have underestimated for a long time how many folks, you know, desire that information. And it's pretty smart. As I was watching some of the episodes, I said to myself, I'm glad he's doing the interviewing, because I could never talk to these people, right? You kind of need people on the same technical capabilities to be able to have those conversations. But I think that's what makes it so valuable.

• 1:09:51Speaker 02

  So there are two sides to it. One, if a person doing interviewing doesn't know much about a subject, then the audience comes for the ride and you learn together. If you already know a lot, you can ask questions a typical journalist would not be able to ask. So there is benefit of this deeper dive, more meaningful information acquisition.

• 1:10:10Speaker 03

  And you guys are in the second bucket, which makes a lot of sense.

• 1:10:14Speaker 02

  So I try to combine the two. I may start asking somewhat questions of general interest, where I already know the answer, but I'm curious about my guest's point of view on it, and we may disagree. And it's also perfectly fine. Always good to have a debate, and hopefully somebody will change their mind.

• 1:10:28Speaker 03

  I could not agree more. How can people help you in your research or kind of in your pursuit of AI safety? Is there anything that the general public could do to either support you and/or help your cause?

• 1:10:41Speaker 02

  So it's good if you have access to people working at top labs, make sure they're fully informed about what they are doing. If you have an option to vote for a politician who understands issues, which I think should be the most important issues for the next election, vote for a person who has at least some policy on advanced AI. Lately, a few politicians have spoken about it. And on a personal level, again, Roman Forum, subscribe, like, and follow.

• 1:11:10Speaker 03

  Amazing. All right. Well, Roman, thank you so much for taking the time to do this. I learned a lot today, and I think you have some very unique views, and have been thinking about these problems for, you know, a long time. I always joke that you've been thinking about AI for longer than most people know how to spell it. So I appreciate all the work you've done, and we'll definitely do this again in the future.

• 1:11:30Speaker 02

  Thank you so much for inviting me. Thank you.