86% of What Coding Agents Do Is Just Reading — Not Solving | Alexander Whedon of Subquadratic

Every AI model in production today has the same hidden tax: doubling the context window quadruples the compute. That's what quadratic compute complexity means in practice, and it's the reason enterprises are spending most of their AI engineering budget on context management rather than on the actual problems they're trying to solve. Alexander Whedon, co-founder and CTO of Subquadratic, joins Craig Smith to explain how SubQ's sparse attention mechanism eliminates that tax, achieving 40 times faster inference and 64 times less compute than standard attention at one million tokens, and what becomes possible when that constraint disappears. The conversation covers striking benchmark findings: 86% of what frontier coding agents do is "read steps," just trying to gather and organize context before the actual problem-solving begins, and frontier models drop well below 50% accuracy on financial document analysis at 500,000 tokens, revealing how asymmetric long context capability actually is across industries. The most commercially important argument in this episode is about enterprise data. Most large organizations are sitting on hundreds of billions of tokens of data they've never been able to put to work in an AI product, told they need a $10 million data transformation project before they can even start building. Alex's core claim is that SubQ's architecture makes that barrier no longer necessary, enabling enterprises to process far more of their data with far less curation, at a fraction of the cost. He closes with what he describes as the most important and underexplored frontier in AI right now: we are still very far from understanding what users actually want from models reasoning over millions of tokens, and the product and alignment work needed to answer that question has barely begun. Subscribe to Eye on A. I. for weekly conversations with the people building and deploying the future of AI.

• 0:00Speaker 00

  In transformer models, the cost of attention grows with the square of the context length. So doubling context makes a compute roughly four times more expensive. And Alex and his team have solved this issue.

• 0:18Speaker 01

  When we've done benchmarking frontier models with Subbench Pro, we found that eighty-six percent of the steps were actually read steps, just trying to do that context engineering before execution review, which was only the last fourteen percent.

• 0:29Speaker 00

  Is this the end of RAG? If you could put everything that's in your RAG knowledge base into the context window, you wouldn't need that more complicated architecture. Is that right?

• 0:41Speaker 01

  Yeah, it is, although I wouldn't say it's the end of RAG. Just because it's too extreme. You have all of these enterprises that are sitting on massive amounts of data and a lot of that data has yet to be put to use in an AI product. They're getting pushed to do like a $10 million data transformation project before they can even start to build a product on top. And value prop that we bring to the table is that is no longer necessary.

• 1:05Speaker 00

  Hi, this is I on AI. And today we're talking to Alex Whedon about his company SubQ, which solves a problem that plagues transformer models. In transformer models, the cost of attention grows with the square of the context length. So doubling context makes a compute roughly four times more expensive. That's what we mean by quadratic compute complexity. And Alex and his team have solved this issue.

• 1:49Speaker 00

  So Alex, can you start by introducing yourself to listeners and we'll go from there?

• 1:58Speaker 01

  Sure. I'm Alex, co-founder and CTO here at Subquadratic. I've been in the space for about 10 years. I started working with language models since before transformers existed, when LSTMs were still the hot thing. And our mission here is to always be the first to make the most computationally memory and sample efficient algorithms and build foundation models on top of those.

• 2:27Speaker 00

  Yeah, and can you explain? You've tackled this quadratic complexity with sparse attention, am I right? And can you explain what sparse attention is in simple terms? And why it matters compared to standard transformer attention?

• 2:49Speaker 01

  Sure. So the quadratic bottleneck of attention is introduced because attention compares two token relationships. So you have 1,000 tokens and there are 1,000 squared two-token relationships, or a million squared two-token relationships for a million tokens. You have this quadratically growing set of math that you're going to perform with an attention. Spars attention says, we don't actually need all of those token relationships. And so sparse attention attempts to find which token relationships are important before running attention on only a subset of the token relationships. And one of the main challenges with sparse attention

• 3:32Speaker 01

  is that there's always been a quality trade-off. As you can imagine, like in theory, if you're not looking at every token relationship, then you could potentially miss some important ones. However, you're also looking at a lot that don't actually matter. And so the challenge of sparse attention has been figuring out which ones actually matter. Most sparse attention mechanisms have used fixed patterns. So based off of the distance between the target token and all the other potential tokens, there's some fixed mathematical formula that determines which tokens should be passed through attention. DeepSeek sparse attention upended this by saying, we're actually going to use a separate prediction model

• 4:16Speaker 01

  to determine which tokens are important. And that worked. And they showed that if you dynamically select the token relationships before running attention over a subset, then you can do so without a quality trade-off. The only problem is that prediction model that selects the token relationships uses full attention. And so you still have the quadratic problem. In fact, that sparse, sorry, that separate selection model uses more compute than the sparse attention layers in the full model at just 52,000 tokens and then continuously grows to an even larger differential.

• 4:56Speaker 01

  And so what we've done that's unique is we have a sparse attention mechanism that, like DeepSeek sparse attention, dynamically selects the token relationships, but does so without the massive cost of full attention and does so without a separate selection mechanism. So the single model does both the selection and generation at the same time.

• 5:17Speaker 00

  And can you give us a practical use case where this quadratic compute complexity makes it too expensive to run the model on something? I mean, code bases seems like an obvious, but are there other situations where you want a very large context window and that it would get unreasonably expensive to do that with regular attention? I mean, regular transformer algorithms?

• 5:55Speaker 01

  Yeah, absolutely. I mean, I think a very significant portion of enterprise workloads fall into this category. And by the way, I will just clarify now that we're not just after the absolute context window size. We're after higher intelligence within that context window, lower cost of context, lower latency for context. And so these are actually four different axes of context we care about: size, intelligence, cost, and latency. And enterprise workloads are dramatically impacted by these. Everybody's making trade-offs. Everybody's trying to reduce the number of tokens that they're using as much as possible. Everybody is doing context engineering that requires a lot of effort

• 6:39Speaker 01

  and also narrows the scope of what that system can actually handle. But some of the top use cases that we see are, yes, code is obviously one. Thinking about code review, planning, and long horizon memory, those are all long context problems. Thinking about the fact that when we've done benchmarking frontier models with Subbench Pro, we found that eighty-six percent of the steps were actually read steps, just trying to do that context engineering before the execution review, which was only the last fourteen percent. So and then financial document analysis is another really big one. Financial documents can be a couple hundred pages. And if you really want to analyze trends,

• 7:24Speaker 01

  then it's helpful to have many of those documents. Over a period of time, we're several different companies. We've, in enterprise, you have hundreds of billions of tokens of textual data. And anytime you want to tackle a problem, it's helpful to have like a broad range of documents. The key point that we're conveying and working on with enterprises is how do we do coarser grain retrieval, more retrieval steps at once? How do we avoid chucking documents as much as possible or make larger, more comprehensive chunks, et cetera,

• 8:08Speaker 01

  to offset the quality loss that we often see with a lot of the context engineering that's done today?

• 8:16Speaker 00

  Yeah, and that quality offset, or you said earlier a moment ago that it's not only the cost, but it's the intelligence within the context. Is that how you phrase it? Because large context windows, many LLMs don't operate optimally within a very large context window. They don't see things or they get lost in the context. Can you talk about that? How do you solve that?

• 8:58Speaker 00

  I mean, even though you can now do very large contexts efficiently?

• 9:05Speaker 01

  It's a great question.

• 9:09Speaker 01

  So yeah, the intelligence within the context window is a very important topic. If we just like take a step back and think about what frontier context has looked like historically, we didn't get frontier models with more than a 256,000 token context window until March, when Opus 4.6 general access, 1 million token general access came available. And so until then, most enterprises were building with a less than 256K usable context window. You see that, so a lot of people that we talked to, like they're using 100,000 tokens or less

• 9:50Speaker 01

  and they haven't yet adapted their workflows or RAG setups to use more than that. Sometimes they feel a quality trade off above that with particularly hard problems. Actually, 100,000 is a fairly common cutoff, surprisingly. In the coding space, a lot of people set the compaction limit to 400,000 tokens because that's where they feel there is a qualitative trade off within a 1 million token context window. Because of this, in addition to intelligence, I would say there's also an alignment problem within the context window that user preferences with how a model should handle, should leverage 1 million tokens are not super robustly understood today.

• 10:35Speaker 01

  There are just not enough enterprise applications built that actually leverage a full 1 million token context window to really understand what that should look like. And so as we push into the multi millions of tokens, this is very much white space, not even just from a technical perspective, but from a product perspective, what do users want? There are all sorts of benchmarks that kind of measure human preferences at short contexts, but along contexts those don't exist. And then finally, I would just add that one of the main, one of the challenges with large context intelligence and user preference matching is the amount of training that you can do at large context. And so that's one of the main reasons that we set out to create SSA in the first place

• 11:19Speaker 01

  was that it is cost prohibitive to do large scale training on large contexts. As far as we know, we are the only people to do robust multimillion token pre-training, which we think differentiates our models. So like the number of times that a model can see how to handle a certain situation at a reason over some context at large scale, as that number goes up, obviously the model gets a lot better and that's really hard to do with quadratically scaling compute costs.

• 11:51Speaker 00

  So go a little deeper on that. I mean, you've trained

• 11:59Speaker 00

  and evaluated up to 12 million token context windows. So explain how you avoid the problems that large context windows create. I mean, you go a little deeper on how the model approaches that.

• 12:22Speaker 01

  Yeah, so I mean, I feel like I kind of buried the lead on that with one of my earlier answers. But I mean, the problem with sparse attention historically has been that they're fixed pattern mechanisms, like sliding window attention, look at the last 1,000 tokens that are closest. And, or before and after, but, but these fixed patterns don't robustly model language. So like, there's no guarantee that the first and seventh token are always going to be relevant to each other. It's highly context or content dependent.

• 13:04Speaker 01

  And so you need a mechanism that accurately models context, accurately determines which relationships are going to be important, no matter what the input looks like. And, and so while DeepSeek sparse attention uses a separate model to do that, reintroducing the quadratic problem, we do not. We have a mechanism that is much more lightweight and cheaper and leverages the existing model that does the generation to do the selection as well. So it's super cheap, super fast, but also dynamic. So every input gets a different attention map.

• 13:49Speaker 01

  And, and then we train, we leverage that, the fact that we can do this dynamic selection to achieve high quality results while also having efficiency differentials. We are around 40 times faster than flash attention before on B200s and B300s. And, and then we're using 64 times less compute, both of these figures at a million tokens. So this means that we can do a lot more 1 million token training with the same compute or 2 million token training or whatever.

• 14:31Speaker 01

  And so this is, this enables us to introduce a lot of problems that models have yet to be trained on robustly. The reality is, is we're training on a lot of data that hasn't been included in training to date and introducing capabilities that have yet to be introduced into the models to date because we have this mathematical efficiency differential that just makes the math of doing this training so much different.

• 14:56Speaker 00

  Just because my audience isn't all ML engineers. So, so the model, you use sparse attention. And you, how does the model decide what tokens to pay attention to? I mean, you were saying that DeepSeek has a separate model that does that prediction, but can you go into how your models with these large context windows then can decide which, which tokens to focus on?

• 15:38Speaker 01

  Yeah, I mean, that's the highly proprietary part that we have openly, but, but I would say, I guess, I could say we creatively use some early information on the inputs in a way that doesn't create all to all comparisons, which is very high level, but I feel that going deeper than that leaves a lot of breadcrumbs.

• 16:10Speaker 00

  One thing I wanted to ask you, and we've spoken a couple of times, but does this, is this the end of RAG then of having a vector database that the model searches? I mean, could you, if you could put everything that's in your, your RAG knowledge base into the context window, you wouldn't need that more complicated architecture, is that right?

• 16:43Speaker 01

  Yeah, it is, although I wouldn't say it's the end of RAG, just because it's too extreme. So the way, it changes RAG. What we're looking to do is simplify the way that people build products with RAG or, you know, agentic workflows. Examples are today, a lot of documents are chunks that don't necessarily need to be chunked. The chunk sizes are very small, 400, every 400 tokens is like super common. And they're like, what if you could have a multi-page, multi-section chunk instead?

• 17:24Speaker 01

  There would be a lot more contiguous context that would prevent misunderstandings. Top K of 10 for the top 10 search results is very common. Well, what if you could pull the top 50? What if instead of doing one type of search, you actually did like 20 parallel searches and dumped all of those search results into the context window? These are ways in which I think we make RAG a lot more, we can enable people to make RAG a lot more robust. If you have 400 million tokens, like there still needs to be some way for the model to process that data. But we're saying like, don't compress everything into 100,000 tokens or less.

• 18:09Speaker 01

  You can now afford to because of costs and latency and then also the intelligence and absolute budget of context. You can now pull a lot more context and do a lot more work with every single step in your agent workflow. Like I mentioned before, if we're talking about coding problems, Opus 4.6 required an average of 66 steps to tackle a few bunch of problems. And, and if we assume that we only needed like the last 10%, then we could have gotten away with like six or seven steps if we had been able to do, if we'd been able to just put all the code in the context window. Or even if we took like three or four steps to, to figure out which parts of the code

• 18:53Speaker 01

  base are more important, being more aggressive with the context in each step. I mean, that is a much faster, much more efficient process that actually uses more token, sorry, fewer tokens in aggregate. So it's, it's more token efficient. And, and then it's not going to miss the details. We do hear complaints about coding agents missing details and, and definitely on the longer horizon. So, so not the end of RAG, but the transformation of RAG as we know it, so.

• 19:29Speaker 00

  And, and you talked about doing multimillion token pre-training rather than just post-training. Can you walk us through how that, how that works and how that changes the model's behavior and practice?

• 19:47Speaker 01

  Um, yeah, so pre-training is the stage of training where you give the model its core knowledge and to some extent, some core capabilities.

• 20:01Speaker 01

  But it's very knowledge focused and so the pre-training simply means you predict the next token over and over again. And, and then the model gets penalized based off of how often that next token matches the actual next token in a corpus. So it's crude and rudimentary, but it's also highly scalable and it's largely seen as a requirement to extract additional capabilities in the, in the post-training phase. And so, um, research has largely suggested that your ability to create value in the post-training phase is limited by the pre-training that you've done.

• 20:44Speaker 01

  So if you haven't, for example, done a lot of pre-training on code, then you're going to struggle to post-train a model to be really good at code. And so that's why it's important that we've done this multimillion token pre-training, because it means that when we get to post-training, there's already a certain amount of that knowledge there, there's a good foundation to build on. And in post-training, we can extrapolate even further. And in fact, this is something we showed in our technical report for our sub-Q 1.1 small model.

• 21:14Speaker 01

  We, we actually have a model variant where we trained the model, we pre-trained the model with 1 million token inputs, up to 1 million, but a lot of 1 million token pre-training. And we showed that that 1 million token pre-training made it so that when we did post-training, largely also only up to 1 million tokens, that the model was able to extrapolate how to do retrieval style problems up to 12 million tokens without ever seeing anything in post-training over a million tokens. So that's the power of pre-training. It really, it enables the model to extrapolate much further than if it's seeing stuff for the first time in post-training.

• 21:53Speaker 00

  Yeah.

• 21:55Speaker 00

  A lot of people right now, particularly with Mythos and Fable out now from Anthropic and GLM 5.2 from

• 22:08Speaker 00

  Zerpu or ZAI, yeah. Many people worry about large context models making it easier to find vulnerabilities in code bases. It is, how do you view that? And, and in, in both offensive and defensive cybersecurity use cases?

• 22:33Speaker 01

  Yeah. I do think that long context windows, long context reasoning capability does help with finding vulnerabilities in code bases, but that's different than finding vulnerabilities in products. So finding vulnerabilities in code bases enables you to be more defensive. Finding vulnerabilities in products puts your product at risk. So two very different things.

• 23:01Speaker 01

  In our case, if you already have all the code in the code base, being able to reason over a lot more of it at once enables you to find vulnerabilities much faster and, and also find vulnerabilities you wouldn't, would not have been able to find otherwise.

• 23:17Speaker 01

  And so we see that as something that's really valuable for people being able to make their products safer and more robust. That is, some of our design partnerships have, have leaned in that direction.

• 23:32Speaker 01

  I don't think that long context reasoning specifically provides nearly as much an advantage for being able to find the holes in a product from the outside. Because they're more about iterative tests. The context requirement is not necessarily significant, but the way this works is software is built layer upon layer of extraction, of abstraction. And so, you know, there, there, there are probably like 20 layers of, of different code components underneath the code that you're writing at any given time.

• 24:16Speaker 01

  And so if I'm trying to hack your product, I'm trying to guess which open, I mean, there, there are other things too, but this is a big part of it. I'm trying to guess which open source software you've built on top of. That, and, and these open source softwares have known vulnerabilities. And so I'm trying to, I'm trying to, I'm testing to see if any of those known vulnerabilities exist in your software. So it's not really a long context reasoning problem. To be honest, I think more than anything, it's actually a knowledge problem. It's, it's an existing knowledge of all the, the existing vulnerabilities. And that's why you've actually seen that people can, can get open weight models to,

• 25:01Speaker 01

  to have Mythos level capabilities for, for hacking, because the model needs to specifically know, to be honest, sometimes, like the number of vulnerabilities that, that lead to most of the threat, it's lesser than some people realize. Because everybody builds on a lot of the same software components, so.

• 25:23Speaker 00

  Yeah, but couldn't this, I mean, to find those, well, you, you said they're known vulnerabilities in open source code bases, but with Mythos or, or, you know, or, I mean, with your model that, that can handle massive contexts, couldn't you then just, all the various foundations that are managing open source code bases, they could just dump it all into your model and, and discover vulnerabilities that maybe they weren't aware of, patch them.

• 26:05Speaker 00

  I mean, is, isn't it advantageous, advantageous for, for the open source community to have solve the context compute complexity problem so they can scan large code bases and, and fix vulnerabilities?

• 26:25Speaker 01

  Yeah, I think that's a great point. I think partnering with open source communities to find and fix vulnerabilities faster and find and fix the vulnerabilities have just been there for a long time where speed isn't the issue, it's the breadth of perspective that's the issue. That'd be a great use case that I'd love to work with folks on.

• 26:47Speaker 00

  Yeah, you've mentioned design partners a couple of times.

• 26:53Speaker 00

  What are the most compelling early design partner use cases you're seeing for a large context? Is it finance, code, internal documents?

• 27:06Speaker 01

  So unstructured document processing and analysis is like the number one right now. And the number two is knowledge-based products.

• 27:19Speaker 01

  Within those two categories, finance emerged as a really important one, just because finance, companies working in the finance space are building such data-intensive applications, they're processing so much data. That's kind of like the core criteria for whether a design partner, early access user is a good fit for us right now is how data-intensive is their application? So finance is a, a really big one for us. We have had a lot of interest on the coding side. We are intentionally moving a little bit more slowly on the coding side because we've kind of realized like how much coding is like a budget game, like how much data budget do you have?

• 28:04Speaker 01

  How much labeled data can you train on? And so that's an area that we're really interested in and are working on, but with a bit of a longer time horizon. And then, and then it's beyond that, it's, it's like general enterprise document processing for knowledge bases and unstructured inside extraction. Those are the main use cases that we're seeing.

• 28:33Speaker 01

  One thing that's great for us is that I, I think unstructured document processing and knowledge-based products are in pretty much every enterprise out there.

• 28:42Speaker 01

  And we're also seeing that because we can work with data that is less structured and we can process more data at once, it becomes a lot easier for an enterprise to get their first value from data. So you have all of these enterprises that are sitting on massive amounts of data and a lot of that data has yet to be put to use in an AI product. Or maybe it has, but in, in a very limited fashion. And they're getting pushed to do like a $10, $10 million data transformation project before they could even start to build a product on top. And a value prop that we bring to the table is that is no longer necessary, at least not at that level.

• 29:25Speaker 01

  Like, we can help you either work with the data with, with a lot less structure or create the structure a lot faster because of the, our ability to process a large amount of data at once with a lot less curation.

• 29:43Speaker 00

  That's interesting. And as we move into agents, and particularly coding agents, increasingly, which is happening, most coding agents use very surgical context management. I mean, they they hop from file to file or section to section. What do you think breaks first when we move to agents that can read everything at once?

• 30:18Speaker 01

  Yeah, I think that will, a couple of things will happen. One is we will have systems that can generalize better, because every time you add a search engine or you know, vector database, which is a search engine, or some conditional logic to route between the steps. This human curation really limits the ability of that system to do a lot of different things. It makes it so that it works within exactly the scope that you've set it out to to be able to to to achieve. And

• 30:55Speaker 01

  and and the more complicated you make it, the less, the more narrow that's going to be. So I think we'll see agents that can do a lot more. We will also see agents that are a lot faster, because, you know, if you're doing 10 to 60 steps,

• 31:14Speaker 01

  it's pretty slow. So I always say agents that are cheaper. We want, you know, a million tokens to feel like 50,000 tokens in terms of intelligence, cost, latency, et cetera, over time. And and so I think we'll see people build a lot more too, to be honest. Like, context engineering step is where the vast majority of human and capital investment is today. And so if we can get rid of that, then we'll see people, like the barrier to entry goes down, people build a lot more things. And

• 31:52Speaker 01

  and when like a single step command can do like 10 steps worth of work, what are those 10 to 60 step workflows going to be able to achieve now? I mean, it'll be a lot more powerful, a lot longer horizon.

• 32:08Speaker 01

  I guess that's that's another point.

• 32:14Speaker 01

  We really care about long horizon agents and

• 32:19Speaker 01

  the ability to manage context, not just through a two hour session, but across, you know, weeks or months of work is something that I think will be critical and will also require much longer context reasoning than we have today.

• 32:39Speaker 00

  Yeah, we didn't go into your background or the founding of the company. And it's interesting. Can you give us that that story?

• 32:51Speaker 01

  Sure. Yeah. So I'm I've been in the AI space for the last 10 years. Most recently, I was head of generative AI at a mid-market in enterprise AI consulting firm called Tribe AI. I did over 40 projects with a series of mid-market enterprise companies with a medium sized group of folks underneath me. And and that's where I got a really good horizontal view of what generative AI implementation was looking like in 2023 and 2024. And that's kind of what convinced me that like it was overdue to move to another model paradigm, that they were bottlenecks in the enterprise implementation layer,

• 33:36Speaker 01

  that we're going to be painful for a very long time unless we change the actual mathematics behind the the foundation models themselves. Before that, I was at Instagram, working on creator monetization products. Before that, I was at Stitch Fix, where I built the first text based recommendation system. Before that, I was at a research lab where I published some research and built our first production products for our sister companies Glassdoor and Indeed. Before that, I was at academia.edu, where I built some of our first AI products for our 200 million user base. Before that, I did research with the World Bank of Blue Cross Blue Shield.

• 34:18Speaker 01

  And so I've done a lot of different things. I'm not a domain expert. In fact, that's one of the things I pride myself in is I'm, I like to look at things from a very horizontal perspective. What's the key value point that's missing across everything everybody's doing? And that's really what we're trying to tackle as a company. It's like, how do we take computational memory and sample efficiency and turn that into downstream business value propositions? Context is the first one, but there will be many more.

• 34:51Speaker 00

  Yeah, well, and you mentioned memory. Are you working on memory?

• 34:55Speaker 01

  Absolutely. So, I mean, we're working on memory within the context of SSA, sub quadratics, price attention, the thing we talked about publicly. We're also working on non-attention algorithms, have been for a while,

• 35:07Speaker 01

  that make it totally necessary to have any form of KV cache. KV cache is this massive thing right now, where at the multimillion token range, can easily use a lot more memory than the model weights themselves. It becomes very hard to host the context without distributing across multiple cloud data center grade GPUs, makes it impossible, in my view, to do long range modeling for robotics on jets and Orens, which have about as much memory as your phone. And so we see this as a critical problem to solve for long context modeling, for general robotics intelligence,

• 35:52Speaker 01

  and just the ability to create much better frontier models than exist today.

• 35:58Speaker 00

  The this long context, you mentioned that

• 36:06Speaker 00

  most enterprises don't even fully use 256K context. As you introduce them to multimillion token windows, what what are you learning about user preferences and behaviors?

• 36:26Speaker 01

  Yeah, that, I mean, that they're not well mapped out and they can vary. So we found, for example, people will tell us that

• 36:42Speaker 01

  on a news article, they'll ask, what are the top three insights from here? And the model will give three top insights that are technically accurate, but they're not the three they cared about. So like, this is the level of alignment that becomes necessary with long context reasoning gets very complicated. So one of the things we learned is that we are so far from from this area of modeling being mature. I think people say, well, the frontier models have a million token context window, but like their ability to deliver what people want at that size is just completely, we are far removed from that today.

• 37:22Speaker 01

  And impromptability is still a bit lacking also for these types of problems.

• 37:29Speaker 01

  And I think to some extent, like this problem can't be solved at the the prompt level, because it's just so complex and nuanced. Like, we do, we do actually need to solve this through training first, where we see a very wide number of problems that a very wide number of inputs and outputs or problems through RL

• 37:53Speaker 01

  that show what user preferences are at scale before this becomes a problem that we can really solve through prompting.

• 38:03Speaker 01

  So that's a super high level insight. Beyond that,

• 38:12Speaker 01

  I think reasoning is pretty unsolved. Oh, here's here's an interesting one, actually. We did some, we created some evals for codebase question answering and financial document question answering. And we ran, we did them at 500,000 tokens,

• 38:32Speaker 01

  a million tokens per Opus 4.7, which is like 800,000 tokens for us.

• 38:38Speaker 01

  And and then as close as possible to 2 million tokens, which obviously only we could do. And so we found on codebase question answering.

• 38:50Speaker 01

  There were like different levels of difficulties, like we could say easy, medium and hard. And on the easy and medium problems, like the models, most frontier models and our model could do pretty well, pretty close to a hundred percent. And there was really not much of a concern there on performance. But it is quite easy to create an eval set that on financial document analysis that is well below fifty percent at five hundred thousand or eight hundred thousand tokens. And so it does seem like, obviously, there's a bias towards code.

• 39:32Speaker 01

  I think that the model companies have spent like more time in some areas than others. And then also, I think there are certain types of contexts that are harder than others. Like, I've been working with financial documents for a long time. I've spent a lot of time working with Edgar data. And there are a lot of things that make it hard. Some of them are intentional. But but the structure of the PDFs can be very challenging for LLMs to handle. Data can be really distributed across several hundreds of pages, up to several hundreds of pages. You've got tabular and non-tabular data.

• 40:15Speaker 01

  You've got tables that are really hard to parse correctly. All these challenges that I think make these problems harder than code in some ways. So I guess the TLDR here is that long reasoning capability today is very asymmetric across verticals.

• 40:34Speaker 00

  Yeah. On the, you mentioned that you give these models the standard transformer models a large context, a large chunk of text in the context window, and ask it to surface the most significant things, and it doesn't come up with what you think is most significant. How does this, is the reasoning stronger with your architecture?

• 41:16Speaker 00

  Or is it that it sees more of the context? I mean, why is it better with SubQ's model?

• 41:27Speaker 01

  So the biggest part of it is the training differential that, you know, we can, end to end, our model training speed is still like somewhere between five to 10x faster at a million tokens. And then as you move to two million tokens and beyond, that's higher. The differential is even bigger than that. And so it's easy for us to do a lot more long context training than other people are doing. And we're also very focused on generating a lot more of that data through both synthetic generation and human labeling. And so, you know, focus and scale is a big part of that. This is phenomenon that we call the bitter lesson in the AI space.

• 42:12Speaker 01

  Yeah. Yeah. And so we, so that's a big part of it. I think there's a theoretical benefit that I don't feel like we've fully proven one way or another, but that is that we are reducing the noise for the model. So instead of looking at a million squared relationships, it'll look at, you know, scalar value times a million token relationships. So has less to consider, less to model on top of. And and so the same way that like RAG reduces the noise and could sometimes lead to better results, I feel like sparse attention could also reduce the noise and lead to better results.

• 42:56Speaker 01

  But again, I don't feel like we've fully proven that one way or another. It is not going to be easy to prove because A, how do you separate the architecture from the training? And B, what evals are we going to use that are actually helpful today? I think we have a massive dearth of high-quality long context reasoning evals.

• 43:17Speaker 00

  Yeah. And if someone wants to play around with this, how do they do that?

• 43:27Speaker 01

  So today, I mean, you reach out to us, we'd evaluate you for a design partnership. The number of, we're not giving broad access to the model yet just because we have limited compute bandwidth ourselves and are trying to prioritize access to people that we think could help us make a better version of the product. And and then, and we are continuously trying to increase the amount of compete bandwidth we have. These additional, these early design partnerships are also helpful for that, too. The market validation enables us to increase our capital budgets and thus compute budgets. So, yeah, I mean, that's kind of the TLDR is like, reach out, we'll connect if it seems like an interesting fit.

• 44:12Speaker 01

  We usually go through an evaluation process, scope a design partnership, and then get started. That'll change over the next couple of months. In the next couple of months, we do plan to have complete general access. And by then, we will have a very different version of the product, too. It will be, our models are continuously undergoing training to respond to feedback that we're getting from partners. So if anybody thinks the model is good now, it's just going to get a lot better.

• 44:38Speaker 00

  Yeah. And when you do have general availabilities, is there going to be a free layer or a free trial? Or how are you going to get people in the door?

• 44:54Speaker 01

  So, yeah, I think we'll have some level of free access for sure. And we'll have tiered pricing and tiered rate limits and all of that. So it'll be pretty standard. There are some non-standard things that we are considering doing. One of them is like a higher output to input ratio. Today, the output token cost is usually three to five x the input token cost. We'd like the input token cost to be a lot more cheaper, oh, sorry, a lot cheaper still to make it, we really want people to feel like the input tokens are free. Like, just consider the context that you need to for your problem.

• 45:39Speaker 01

  So I think we'll see higher ratios there. And then, secondly, we've hesitated saying this a bit, but we've considered, I think our pricing curve, as your inputs grow, could look different than than others because our internal price economics look different for it. The external pricing for that could also look different. Like, yes, that's all I'll say for now. So.

• 46:05Speaker 00

  Yeah. And I have to ask, you know, there's a lot of talk about using AI for scientific research. Did you guys use any models to design this? I mean, did you talk to, you know, Claude 4.8 or now, I guess, Fable 5? And, you know, how do we solve this problem and go back and forth with a model? I'm curious how companies like yours are doing that architecture design, whether they're

• 46:50Speaker 00

  tapping the knowledge base of the model or whether it's all, you know, iterative trial and error among humans?

• 47:03Speaker 01

  Totally. So, I mean, we're definitely using LLMs in the process and always trying to as much as possible. I don't think you can get there with just an LLM. They're not enough today. I don't think they're creative enough either. But they're definitely super helpful. And I will say, sometimes I'll just open-endedly ask an LLM to come up with an idea, and it comes up with something I hadn't thought of. So they're super helpful for that. But.

• 47:36Speaker 01

  But we've been talking from probably like somewhere between a year and a year and a half about how to get to auto research for algorithms before, you know, way before auto research was termed, way before RSI was termed, recursive self-improvement. We've been trying to build that product. And so I want to be the first company that creates this high-quality flywheel of a model proposing architectural variations, creating the experiments to validate or invalidate those variations, running those experiments, looking at the outputs, coming up with what the idea is for the next set of experiments and architectural variations,

• 48:20Speaker 01

  until it comes up with an architecture that looks vastly different and is vastly better on the axes that we care about. We will be investing into that very seriously, so.

• 48:33Speaker 00

  Yeah, that's interesting. And looking ahead, what's the most exciting frontier for you? I mean, at SubQ, is it bigger context? And not only at your company, but in across AI research in general? New architectures beyond attention or something else?

• 48:56Speaker 01

  I mean, yeah, a lot of different things. Obviously, I think the auto research thing is really exciting.

• 49:01Speaker 00

  Yep.

• 49:04Speaker 01

  I think that, but again, it's not about doing the pre or post-training better, which is where most people are focused. It is about the algorithm itself. We want to move from like shifting the algorithmic paradigm every nine years to doing it every 12 months. So we want to replace transformers or attention entirely in the not too distant future. And then whatever we replace that with, I mean, we know what we want to replace it with. But when that happens, we want to replace that again within 12 months and just shorten the time as much as possible. So that's, that's interesting.

• 49:48Speaker 01

  I think that from a go-to-market perspective, we're really excited about working with enterprises on long context reasoning tasks, long horizon agentic tasks as a secondary priority. And then in the somewhat longer term, though there is active work being done on it, we do care about robotics and see that as highly intertwined with our current work, actually. Like, the reason why the Figure AI demo was on conveyor belt flipping boxes is because it requires a 15-second memory span. But if you think about asking, if you think about asking a robot to do the laundry,

• 50:31Speaker 01

  the robot has to remember the roadmap of the house to be able to navigate. It needs to remember how to open the laundry machine door. It needs to understand, it needs to know what cycles are appropriate for which, for which clothing items. It needs to remember what these preferences are for how to fold each of the items. It needs to remember where the clothing goes. And then, and so the amount of context that's required there is significant. And then additionally, the, we are in the same place for robotics intelligence, as you were for language models in 2019,

• 51:11Speaker 01

  where we train models for specific tasks with thousands of task-specific samples that were hard to, hard and costly and time expensive to curate. And, and so I think we need a 2020 LM or Few Shot Learners GPT-3 moment for the robotics space. But if you think about providing a 10-minute video sample so that the model can learn in context how to achieve a task without being trained on that task specifically, that is, that's like a 4 million token input. And people are struggling to do that in the cloud.

• 51:52Speaker 01

  How do you do that on a phone worth of VRAM? That is, that requires like 100x plus, a 100x plus reduction in memory usage. And that's a problem we're thinking about a lot. So that's a pretty exciting frontier for us. There are a lot of exciting things that we're doing here. And some of the stuff is, you know, stuff we're trying to put out in the next six months. Some of it is things that we're trying to put out in the next two years. So we've got a longer-term roadmap. We're not just thinking about incremental improvements in the industry.

• 52:34Speaker 01

  We want to figure out what problems are going to matter two to five years from now. How do we, like, take a shortcut to get there? So.

• 52:43Speaker 00

  And how big is the team?

• 52:47Speaker 01

  So we have just under 50 people today. We've grown a lot in the last couple of months. I mean, the fundamental research that we did, that we've announced, was done by just a few people. But we've been sitting on a lot of research for a while. That research was done in 2025. And then production, or productized this year, but it was done with a lot fewer resources and people than we have today. And so, and we've done a lot since then. And, yeah, I mean, it's a little under 50 people today.

• 53:24Speaker 00

  Okay. Well, this is all fascinating, and I'm glad that we've met. I'm certainly going to be following you guys. If listeners want to follow what you're doing, is it subq.ai? Do you have a blog or something there that people can follow?

• 53:48Speaker 01

  Yeah, subq.ai is good. We've released a couple of blog posts. I wouldn't say that we're prolific bloggers yet. We do want to try to push out more, but it does take some time. I am trying to post at least once, if not a couple of times a week on Twitter. And I do want to start going a little deeper on some of our product and technical and research vision here in some of those posts. And I'll call out different things every once in a while. So that could be an interesting place to follow us as well. We have our official Twitter page and, you know, may see some of our other employees being more public as well about some of their work. So social is going to be a pretty good place to stay on top of what we're doing as well.

• 54:33Speaker 00

  Yeah. Okay, Alex, well, this is really interesting.