The Hardware Bottleneck AI Can’t Fix

Software engineering has developed powerful tools for observability, data management, and continuous testing, but hardware engineering has largely not kept pace. The feedback loops, tooling, and infrastructure that software engineers take for granted simply do not exist in most hardware programs. Nominal https://nominal.io/ is a data platform built to help hardware organizations move at the same speed as software teams. It manages the hardware data supply chain end to end, from ingesting high-frequency sensor data off physical assets to enabling real-time control room monitoring, post-test analysis, and simulation correlation. Jason Hoch https://x.com/jrshoch is the co-founder and CTO of Nominal, and he has a background spanning distributed data systems at Palantir and cloud infrastructure at Vercel. In this episode, Jason joins Kevin Ball to discuss why hardware engineering has lagged so far behind software in tooling and observability, the unique data challenges of working with high-frequency time series sensor data, how Nominal handles both real-time control room workflows and post-test analysis, why AI agents are transforming software development but have not yet made the same leap in hardware, and what it would take to close that gap. Sponsorship inquiries: [email protected] https://cms.megaphone.fm/organizations/619b680e-d813-11ea-9750-e7ecac406436/podcasts/e60108fe-e328-11ea-b2b6-1348249f900a/episodes/3913370a-bc44-11ed-ace6-8b1f96d82bba/[email protected] The post The Hardware Bottleneck AI Can’t Fix https://softwareengineeringdaily.com/podcasts/the-hardware-bottleneck-ai-cant-fix/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  Software engineering has developed powerful tools for observability, data management, and continuous testing. But hardware engineering has largely not kept pace. The feedback loops, tooling, and infrastructure that software engineers take for granted simply do not exist in most hardware programs. Nominal is a data platform built to help hardware organizations move at the same speed as software teams. It manages the hardware data supply chain end to end, from ingesting high-frequency sensor data off physical assets to enabling real-time control room monitoring, post-test analysis, and simulation correlation. Jason Hock is the co-founder and CEO of Nominal, and he has a background spanning distributed data systems

• 0:44Speaker 00

  at Palantir and cloud infrastructure at Vercel. In this episode, Jason joins Kevin Ball to discuss why hardware engineering has lagged so far behind software in tooling and observability, the unique data challenges of working with high-frequency time-series sensor data, how Nominal handles both real-time control room workflows and post-test analysis, why AI agents are transforming software development but have not yet made the same leap in hardware, and what it would take to close that gap. Kevin Ball, or K-Ball, is the Vice President of Engineering at Mentos and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup,

• 1:27Speaker 00

  and organizes the AI in Action Discussion Group through Laten Space. Check out the show notes to follow K-Ball on Twitter or LinkedIn, or visit his website, kball.llc.

• 1:49Speaker 01

  Jason, welcome to the show.

• 1:51Speaker 02

  Thanks for having me, Kevin.

• 1:52Speaker 01

  Yeah, I'm excited. It's gonna be fun to talk with you because the subject area of hardware and hardware testing is like a slant for me. It's like real close to what I do, but it's not what I do. So I'm excited to dig in. But let's start with you. Can you give me a little bit about your background and like how you got into what you're doing today with nominal?

• 2:10Speaker 02

  Yeah, I'll give you kind of a longer story, which is that I started my life as someone who was more of a mathematician than an engineer. I think software was a way for me to like enter engineering. And I got like, I fell in love with distributed data systems. I built a lot of software at Palantir early in my career. Also fell in love with working for like really technical users, people who had expertise in something, but not necessarily software and data systems. And nominal for me is like the culmination of some things that I'm really passionate about. So it's growing software teams and products from scratch, but also with like big ambitions, meeting those really technical users. In our case, it's like hardware engineers, people who have obsessed over electrical systems, aerospace systems, and then like frankly, just solving like really hard but fun like software problems. So

• 2:54Speaker 02

  full stack up and down, like how do you store petabytes of high frequency sensor data all the way up to like the minutia of the interactions that people expect out of our front end products? Because you have to consider those who are thinking about how the data is stored and moved around. So I've been doing that over the course of my career. I love tooling. I love infrastructure. I've been lucky to be part of some like hyper growth organizations. I'd spent some time at Versale working on their like cloud infrastructure and how do you make like very low latency web experiences for people across the globe? That team was very fun. It was four people when I joined and we grew to 40 in under nine months. So in many ways, not my first career, but gotten to do some different things over time.

• 3:31Speaker 01

  Awesome. So let's maybe start with a quick overview of what nominal does for folks and then we can dive into the technical guts of it.

• 3:39Speaker 02

  So we try to help hardware organizations move at the speed of software. And we think that the right way to do that is to build a connected suite of software products specifically focusing on managing data. And we talk about the hardware data supply chain. So there's a physical sensor somewhere in the world and fast moving, modern hardware programs often need to get that data in Like many, many different eyeballs and they're working with like really high frequency multimodal data. So you can imagine like a million data points per second are being produced on something like a test aircraft. And you want to correlate that with video or you might have some sensors or other data sources that are on the ground while the airplane is in the air. And getting that all to work well together is like a software problem and it's a human and a process problem.

• 4:24Speaker 02

  And as software engineers, we've built like amazing tools for ourselves. We're very used to like an elastic cloud where you can run like infinite C I C D. And the observability that we are used to is just like not there in the hardware world. Some companies like SpaceX and Tesla and then Anduril, one of our customers have like done really good stuff here. But there's this like wave of like smaller companies that are trying to be really innovative at hardware and move fast. And they really need help from software engineers and people who've spent a lot of time there.

• 4:52Speaker 01

  You know, most of the folks listening to this are, we are software engineers, right? And we know maybe what this looks like in the software industry, but have no idea what the state of the art is or what the norm is in in hardware engineering. So when you say like helping hardware move at software speeds, like what is the traditional hardware speed look like? And what are the pieces that go into that?

• 5:12Speaker 02

  Yeah. So when my co-founder Bryce and I were getting to know each other and he was teaching me about like what he had done NASA mission operations. So like we have sent a probe, we've spent over a billion dollars of taxpayer money to Jupiter. Like, what is it like to deploy software procedures to that probe? And more importantly, like test them ahead of time to make sure that you don't like crash or destroy the mission. And my mind was like immediately like I started geeking out because I realized that when I think of C I C D, I think of these like, again, like you're doing a unit test. Maybe a more complex integration or end to end test would take minutes. But when you're talking about like something like a satellite, an airplane or a spacecraft, you might spend a million dollars on a test bed. Like a physical table where you're like arranging batteries and wires.

• 5:55Speaker 02

  And you need to pre-test before you test on that test bed. Because if your test bed breaks, you might only have one or two of them because they're so expensive and your whole organization grinds to a halt. And so like the hetero heterogeneity, I always have trouble saying that word, but it's like, it's not homogeneous the way you. Kind of do this, you know, something like a CI test pipeline. And so immediately there's like a lot of need for like interesting tooling there. But just to kind of like ground the conversation, I always like to talk about test aircraft. It's really easy to talk about, but you know, a lot of our customers, they fly something in the air. It lands, you need to get the hard drive off of the airplane. You need to somehow get that data to an engineer who designed the system. If there was an issue and you need to kind of debug it.

• 6:35Speaker 02

  And you know, some of our customers, that process could take on the order of like literally like a day or two. Versus we're used to that feedback cycle being like so much tighter as software engineers.

• 6:45Speaker 01

  Yeah, it's wild to think about like to get your test data back. You have to literally transfer the hard drive. And then to run another test, you're either flying a plane again, or you're you're configuring some sort of hardware system. Okay, so with that as like our grounding statement, right? Hardware has these like physical constraints. What can you actually do with software to make that better?

• 7:11Speaker 02

  Yeah, so again, just to bring us back to that example of like the airplane lands and now you're getting the data off of it. A really common stumbling block we had seen is that the schema for the data that you're taking off of the hard drive. And now maybe trying to get into some kind of standardized long-term format. It can change as the software you're deploying to the aircraft itself changes. And in the field, you might have someone who's really tired. They're trying to get home at the end of the day and they're not a software engineer. They're not even some an engineer who designed the physical system. They're an operator or technician. And they might forget to change a config value in their ground control software. When they let's say they downgrade the software because there's an issue. Now you need to downgrade the configuration.

• 7:54Speaker 02

  And if you don't, the data gets garbled and all of a sudden like, you know, you're back to this like really hard human process of ever trying to recover it. That's the kind of stuff that we kind of realized was happening in some of our customers. Like, one of them said, hey, all of the data ends up in this like network storage, like help us dig in there. And we realized that only 10% of it even got to that point because of these kind of like earlier pain points. But like, all of that is something that we think that we can help with. So in that example I just gave, we actually built a better software tool for the test operators. And then they didn't even have to think about this, but one of the side effects was that that config value was getting automatically updated because we were syncing the version that's running on the aircraft with the version of the partial that was running

• 8:39Speaker 02

  locally. And then instead of them having to upload a file and kind of do this process, like, again, that was happening in the background. That's like a more clunky example, I think, same organization, but now let's say they're installing a Starlink receiver and transceiver on the aircraft. Like, all of a sudden, they're going from a, the data is getting ripped off of a hard drive at the end of the flight to it is streaming, welcome to 2026. But that can be a really hard architectural change to navigate if your software resources are focused on the guidance and navigation and control of an aircraft, not the data processing. But we helped tons of customers with that transition. Our architecture is designed to handle it. And it's a fun and interesting software problem to solve. But that's the kind of thing that we can work with them on.

• 9:20Speaker 01

  Yeah, it's fascinating because you're almost having to be like vertical integration across all these different levels of like from the just making it easy to collect the data at all, right? The automated configuration change, suddenly like the data is actually getting through without a person doing that to like, hey, you can actually stream this data. Like, let's help you handle that. Completely change the way you think about data. So I guess the question would be then, what are the problems that you end up having to, like, you said, we're going from essentially batch hard drive data to streaming data. You have to support that whole gamut. What does that end up looking like in your architecture? Like, what did you have to do to do that? And what does the end user perceive as they make that transition?

• 10:05Speaker 02

  So I'd say that as with any startup, we've felt like over time had to pick and choose which things we handle really well and which things we kind of do a passable job at so that we earn the right to do like a better job down the line. So our original streaming architecture, like I'm going to be honest, was like pretty weak. We kind of built a proof of concept that we could bring to our customers and and open their eyes to like, hey, you are thinking of post-test data analysis in a certain way, but we actually think it could be this new world where you're kind of doing the analysis on the fly. You're some of it's automatic while the aircraft is in the air. And then one of our customers said like, we actually want to have nominal shown in our control room and it's going to be like a human is looking at a dashboard and there's

• 10:49Speaker 02

  a aircraft and there's a person in that aircraft and we want to know very, very quickly if the safety check is failing. And we searched on that as an engineering team because we were pretty confident we could get the latency to be within those bounds. And I guess it was a happy accident that the more analytical user interface we had built was other users were starting to look at it and seeing it as more like a monitoring dashboard. And I would say like now we're in a place where like those two use cases are both supported and like very crucial to our customers. So again, it's like this, imagine you have a control room, like you could think of it like Houston mission control, like you're launching something or there's some kind of test and there's a lot of humans looking at a lot of screens. And then there's the very kind of like you're a propulsion engineer who's digging through hundreds or thousands of flights, trying to find kind of statistical patterns.

• 11:34Speaker 02

  But from an architecture standpoint, like we've had to kind of invest in like, you can think of it as like a hot path and a cold path that need to stay in sync, like they can't be telling different stories. But that really, really low latency pipeline and the more cold storage or I need to do OLP style analysis, we support both of those and are trying to almost abstract away from our customers and our users, the fact that from a software standpoint, those are really very different. We want them to write the logic once and be able to kind of pivot to the different use cases seamlessly.

• 12:06Speaker 01

  It's interesting because like in one of those cases, you are in a traditional development QA cycle or postmortem cycle where you're like, something went wrong or something happened. I want to look at it differently. And the other one is sounding to me like live observability. Like, let's have health dashboards. Let's have, you know, go no-go decisions happening.

• 12:25Speaker 02

  Yeah, it's crazy the amount of data that people will cram into one nominal screen. Like, we've had to adjust parts of our user interface because we didn't necessarily predict like, oh, it looks more like an Excel spreadsheet than a web UI. But it's just because these systems are really intense and the data density is like, becomes very valuable. It's kind of a classic lesson in kind of like user interface. But if you have an expert user who's using this thing day in and day out, it's very different than like, you know, I'm designing a web UI that someone might only, you know, there's a funnel that I need to get them through and that's the only time in their life they're going to perform it. And then I think just like to highlight some of the software fun of this is like, we deploy off-cloud. So like, there's customers where they need everything to be running locally. And I can talk about real-time. That's like a interesting

• 13:08Speaker 02

  topic for us to get into later. But without getting into real-time, like for these low latency control room workflows, you might not want to have a network hopped AWS involved. But there's customers that, you know, they're using Starlink. They don't want to necessarily administer their own stack of data software. And so they are using AWS. And for us to get data going to AWS and then rendering back into a React application, all with like very high volumes of data with like tight SLAs around latency, like that's a really hard software problem that we've invested a lot in. And we think it makes sense for our market.

• 13:40Speaker 01

  So kind of going into that, you know, the volume of data here and thinking about, well, I guess maybe first, let's start with like, what does the data shape look like? Right? Because when we're talking about observability, this isn't stack traces and like post hoc things. This is something else going on. So what is the shape of the data you end up working with mostly?

• 13:58Speaker 02

  Yes, if you think of a satellite or an aircraft, especially one that's in test operations, there could easily be 10,000, 20,000 sensors on that asset. And each sensor might be kilohertz, maybe 10 kilohertz. Sometimes we see megahertz sensors. So that's 1,000, 10,000, or a million points per second. And depending on the test that's being performed, you might never want to drop a data point. So the example I always give is like, if you're building a rocket engine as a startup, you know, you might spend years and tens of millions of dollars getting to the point of doing your first rocket fire test. And even if that rocket fire only lasts eight seconds or something, but like, you never want to throw any of that data.

• 14:41Speaker 02

  That's your entire company is essentially that data asset that you got from that test. And that is super different from software observability where you tend to forget everything. A week later, you're aggregating things into 10 or 15 second buckets and getting, you know, some kind of stats on them. You're thinking about things like P99. I always joke that like, as the software engineers trained to think of my infrastructure like cattle, not pets. But for our customers, everything is a pet, kind of like they will have like one or two aircraft that the entire company is oriented around until you earn the right to kind of scale up production. But yeah, so it's tons of time series data. That video feed can be really important. And doing frame precise correlation between video and time series is actually like really, really hard. That's actually, I'll be honest, we like underestimated the challenge of that at one point in our journey.

• 15:26Speaker 02

  And I'm like very grateful to our, like we have wonderful software engineers in Ominal. And, you know, some of them became experts in like video encoding and like, how do you actually build a video stack? Like a lot of open source software is not designed for that frame seeking use case. It's designed for like social media video playing or something like that. So that's the types of the, that's the shapes of the data we see. And then I'm, I'm just trying to think of like, I already kind of talked about this like network issue, but like you might have radio data, you might have the data that is coming off at the end of the test off of a hard drive. You might have the streaming data. Getting those all to kind of like tell the same story at the end of the day can be a challenge.

• 16:03Speaker 01

  I want to dig in a little bit to what are the differences implied by these companies that are working with pets rather than cattle, right? Like where everything is tied into this one physical thing, or two or three physical things that may be subtly different, right? They may be nominally working towards the same endpoint, but like each one has substantial changes. So like, are there versioning implications? Like, what does that do to the development process?

• 16:29Speaker 02

  Yeah, one of the interesting stories I heard recently is this one company, when they detect a problem in a component, they want to trace it back to which 3D printer printed that component. And they literally named the 3D printers. And it's because the other ones printed by that printer might have had issues around that time. And so it's this really interesting cataloging problem. And at nominal, we talk a lot about the hierarchy of assets, but you might have two aircraft, and each one might have two engines. And at some point, you might actually like swap out one engine from one aircraft to another. And I'm used to thinking about Kubernetes and your nodes and your pods and things kind of get shuffled around all the time.

• 17:10Speaker 02

  But like, understanding which engine, was it engine two or engine three that was on the left side of aircraft 100A last week, like that kind of stuff is make or break. And it can be really, really hard to manage. And I think some organizations, you know, they do have to figure out how to treat some things more like cattle, like pets for their own sanity. But the dollar numbers are just like so different, right? If you launch a satellite and it fails, like that's months of timeline and tens of millions of dollars. And so you're, you're willing to double check things a lot more than I think people in the software engineering world are used to.

• 17:44Speaker 01

  Cataloging is kind of an interesting domain. So is that something that you do as a part of your software as well? Or you provide them tools to integrate their catalog with the test results? Or like, how does that end up working?

• 17:55Speaker 02

  Yeah, one of the things that's really crucial to our data model is this concept of events. And so we are basically through our user interface encouraging people, if they identify a region of time where something has happened, to tag that as an event and then it's available to anyone else who's looking at that. They could be looking at a different subsystem, different set of sensors, but still be aware that something was going on in the overall system at that point in time. And then a lot of these organizations, you know, you, as the responsible engineer who's designed a system, a test happens, you want to be like yourself eyeballs looking at the data feeds to kind of understand, did things perform as expected or is there something anomalous happening? And then over time, you want to kind of like encode those into automated checks. So you're kind of like writing code or lightweight logic.

• 18:39Speaker 02

  By the way, like a story I always tell is that, you know, from the earliest days of our product, you could take these raw sensor feeds, right? I was saying kilohertz sensors, you had 10,000 of them. And you're doing math on them, like these people, you know, they do physics every day. I haven't done physics since I was in undergrad and it makes me sad, I miss it. But I asked them, this is a satellite customer that we were working with early on. I asked them like, you know, what, what type of mathematics do you need to do in our platform? Like we support obviously basic arithmetic, but we're trying to understand how wide our library's scope needs to be. And they were like, oh, like nothing too complicated, just quaternion operations. And they weren't being- Oh, is that all? That's all, yeah. Just the, you know, and it makes sense if you're doing guidance navigation and control for a satellite, that's kind of the table stakes.

• 19:22Speaker 02

  And if anyone has used MATLAB, like, you know, that is a useful thing to have in the back of your mind. We were thinking about the capabilities of our platform. And our goal is not to 100% replace MATLAB in an organization. Like there's always going to be certain types of modeling or calculation or custom code that our users need to write. But, you know, maybe the one person who's writing that MATLAB script can package that up and it can be running in like orchestrated by our platform. And so the other 200 people who might need to take advantage of that logic, they don't themselves need to open up MATLAB or even, or definitely not need to understand the code. I forgot the original question.

• 19:57Speaker 01

  I was just looking for like how much of the categorization, so it sounds like from event standpoint, right, like you can correlate events, right, which makes sense, right? If you're looking at time series data and you say, oh, something went wrong in, I don't know, this engine over here, that's probably going to impact a bunch of your other sensor time series. And so you want to be able to see that. But you also mentioned this example of tracing back the hardware to, for example, the 3D printer that printed it. It made me think of data lineage problems, right? It's not a data lineage problem. It's like a component lineage problem, right? This piece was printed here and assembled here and connected here and like being able to track all that back. So I was curious if that's something that your, your software is supporting, or that's a third-party thing that they're doing independently, or how does that work?

• 20:41Speaker 02

  You're right on the money because I, I always talk about data tagging as well. So yes, it's cataloging, it is tagging, it is, it is very similar to data lineage. You might have, you know, your left wing and your right wing, or you could, you will have these arrays of sensors that are themselves like modules that you're printing multiple times onto your large system that you're developing. And you want to kind of develop logic for checking each of those subsystems. And then you want to repeat that logic, kind of like a software program. But doing that is really hard because you have to have a schema to your sensors that allows it to be possible. That schema will change over time. And it's the types of things that, again, like people who are building for a specific software system are used to solving those problems. But we're trying to build a generic platform

• 21:25Speaker 02

  that can work for someone building an aircraft or someone building a nuclear reactor. And so we think about, you know, tagging of time series data in such a manner that you can make those composable, you can have logic, you can compose it, and then you can apply it to different things. And we think the cataloging is really important. And some of the work we do with our customers is helping them, whether they're like a five-person startup or a 5,000-person organization that's starting a new program, like, how do you be thoughtful about that cataloging from the beginning? Really powerful example, but I think is really grounding is if I am doing a test and something goes wrong, I want to compare it to a simulation that we ran that's supposed to be corresponding with this test. How do I summon that data and like overlay them on top of it?

• 22:07Speaker 02

  Each other really easily. If you've tagged your data well, that's just a couple clicks. If you haven't tagged your data at all and it's just kind of in a big soup, you're kind of writing a lot of custom SQL. And that's what we're trying to get our users away from is, you know, the same way that I use Datadog and the Datadog agent is cleverly tagging many aspects of my data based on the node and the pod and the other infrastructure things that it can introspect. We would love to get our users to the same place. And so for us, that means investing more in the ingestion agent code. That's frankly like running closer to the edge or even like on hardware itself to try to make that a zero cost abstraction for our hardware engineering users.

• 22:45Speaker 01

  That's super interesting. So two different dimensions. I want to go down on this based on that. So one, I'm going to go down a little bit is, so you mentioned using simulation and comparing simulation and hardware. So how, and that, that's, I think, a space that I think it was last year I was at a panel that was hardware man, or it was automotive manufacturers and they were talking about this was one of Tesla's big advantages is that they have all of their stuff in simulation and they can do it. Now the traditional manufacturers are trying to get more and more of their things in simulation environments so they can iterate and improve faster there. Is that a use case that you are directly supporting? So you're doing your test infrastructure for here, like get the outputs of the simulations and then do it with your hardware stack, the live version later? Or is that, once again, like an integration point that you're integrating out to something separate?

• 23:31Speaker 02

  I think for us right now, it's an integration point. So we don't claim to be experts in simulation, but we really do want to make simulation like much more leveraged within our customers. Like if they are running simulations, so many of them run into this kind of like chair swirl problem of like, I need to have three tools up to be able to make the correlations that I want to. Or worse, I need to be like reaching out to another team. I think the world is changing and we, you know, 25 years ago, like true hardware innovation was like pretty localized to large conglomerates, traditional primes, companies like General Electric that had massive, massive resources. And it made sense for those companies to develop over the course of 30 or 40 years, like proprietary best in the world simulation software.

• 24:14Speaker 02

  But as more of this innovation is moving into smaller, more Insurgent companies like they need to grab some stuff off the shelf and try to piece it together. I think that is itself really frustrating. So where we're trying to, we're trying to meet people as close to where they're at as we can. Hey, if you're using the simulation tool, let's figure out how to get that data in nominal. Like I was saying, tag it and catalog in such a manner. To us, it's just data. Like it could have come from a simulation, it could have come from a hardware test. But we'll oftentimes talk to leaders of test organizations that have this dream of like how we're always simulating and if something goes wrong, we're going back and like learning about how to improve the simulation. But their organizations can really easily not meet those aspirations because of tooling fatigue. So that's what we're trying to solve.

• 24:56Speaker 01

  That makes a lot of sense. The other direction I wanted to go down is you talked about having agents running kind of at the edge at the sensor. And I know in sort of software cloud observability, oftentimes a lot of what those agents are doing is filtering out data that is not relevant to send upstream. So they reduce the quantity. But to your earlier point, right? Like the amount of data that you want to filter in a hardware context may be much smaller or maybe very different, particularly, you know, I love your rocket ship example, right? You have eight seconds of data and that is the value of your company right there. That's everything you've invested to get. So you don't want to throw that away. So what exactly are those edge agents doing? Are they still doing some amount of filtering? Or like what happens in that case?

• 25:37Speaker 02

  Yeah, they are still filtering because of like network limitations and the fact that like software is not 100% reliable. But they are then having to save the data locally and then figure out a way to kind of like backfill later. So that's something, you know, we have a customer right now, they're doing like really intensive testing in a network constrained environment. And it used to be that there was massive latency between HQ headquarters reviewing the data and what was going on at the test site because of that network limitation. And we have now made it so that, like I said, the logic can be composed in one place and it can be pushed to the network constrained location. So some of the checks can be happening automatically in a very low latency manner. And then the

• 26:19Speaker 02

  data is getting uploaded at the rate it can. But there's this like mesh concept of you might have your HQ and your test site each Have a database and they they eventually get in sync with each other. You know, again, to use the rocket fire example, the eight seconds, tons and tons of data, but they're not doing that constantly. They might only be doing a few times a day. So if you amortize kind of data upload over long enough time horizons, like you can catch up. But to your question, it's like people who have thought about things like back pressure and distributed systems, super, super relevant for use cases. You oftentimes are having to prioritize between safety, critical information, high priority observability, and then like kind of everything else. And so there's like a hard drive that you're you're having to store to and then catch up later.

• 27:01Speaker 02

  It's not dropping the data. It's just figuring out how to handle it, given the constraints.

• 27:06Speaker 01

  Nice. I'd love to dig in a little bit to your topic of kind of the way that hardware innovation is shifting. First off, just like open question, you said, you know, 30 years ago was all these big companies. Now it's much more insurgent. What are you seeing in terms of hardware innovation and the ecosystem? How is it changing what's happening there?

• 27:25Speaker 02

  Well, it's really interesting because I want to give some credit to some of the larger incumbent players. But like the Air Force being one of them, we work with the Air Force. They employed 20,000 test engineers. Speaking of pets versus cattle, like as taxpayers, we are like all shared owners in arc jets and wind tunnels and these like they're literally, they're called the national labs. There are these assets that are crucial for testing. And whether those can operate well or not, you know, kind of makes a difference between the US falling behind or staying ahead of certain international competitors. So that that is still a thing and like us investing and partnering with those organizations as the software and tech industry, I think is really important.

• 28:05Speaker 02

  But then we have also tons of venture capital dollar and tons of brain power, like trying to do things in new ways. And frankly, like SpaceX has kind of trailblazed here and shown that it's possible. And what's both amazing and terrifying is that these startups are trying to go even faster than SpaceX. Like they're trying to do more with less than ever before. So it's aggressive timelines, it's lean teams. And I think that's part of why we started nominal is like if every single one of these companies is going to be finding themselves having to set up AWS infrastructure. And then build some custom tooling on top of it. That's just not what makes their beer taste better. Like, we should be partnering with them and helping them do that. And we're seeing that happen. So like, we have startups where, and this was not my prediction when I started nominal, but,

• 28:49Speaker 02

  you know, they want to use nominal and they reach out to us and we get them spun up and we send them some documentation. And they're ingesting the data and running without us even having to go on site, which is amazing. And, and these are seven or eight person organizations. Like, we have these metrics where we keep track of like, you know, what percent of the engineers at the company are like logging into nominal on a given week. And for some of the startups, it's, it's 100%. Like, the CEO is logging in because he or she wants to understand how their system is performing week over week. And then I think it's incredible. Like, I don't know if, when I was growing up, I would have thought of a nuclear reactor innovation being able to be carried out by essentially a startup, but we're seeing that more and more. It's awesome.

• 29:28Speaker 01

  No, it's fascinating because I feel like, you know, in my, my career, right, roughly spanning since, you know, early 2000s until now. We went through a period where it felt like all of the fast innovation was happening in software. Software was changing quickly. There were software, software, software. Hardware was slow, moving slowly, not that much change. There was change, but it was like, oh, my car got better miles per gallon now, or things like that. It's not super visible. And then it definitely feels like in the last few years, suddenly we have got private space companies and we have all this like drone changes and all this other stuff going on in hardware that is, it's kind of wild.

• 30:05Speaker 02

  It really is. And we have to ground that in, you know, there's still different constraints building and innovating in hardware than there is in software. And that's one of the things that I get nervous about sometimes, especially in the last couple of years, where software engineers are having agents write code and the agents can run these evaluation loops and there's this feedback loop that's, again, it's just like so fast and so intense. And a lot of times the supply chain is the supply chain in the world of hardware and you can't shave off these kind of months other than by kind of designing things well from the beginning and having an organization that is geared towards iterative development and not kind of like waterfall style development. So if you look at a program like Starlink, they have multiple generations of Starlink.

• 30:47Speaker 02

  Design orbiting the planet simultaneously. They're constantly getting data off of the satellites and feeding them into their manufacturing improvements. And that's the way kind of like every mature hardware program like needs to shift to. You referenced Tesla and their simulation, like the amount of effort that Tesla has put into their internal data cataloging. I think would like blow other people's minds, you know, Apple, not an organization that's super public about the way they handle this kind of stuff, but absolutely massive investments in kind of test infrastructure and test data management. But like, the truth is that, I don't know if it's like because of 3D printing or just the fact that like so many people have like grown up inside SpaceX and now they can go off and build their own companies. And there's this kind of like culture of confidence in doing things more quickly.

• 31:28Speaker 02

  But yeah, we're kind of in like a golden era of of that innovation. And as a software engineer, I'm a little bit, I get phoned with sometimes. I still love building software. And so I I joke that nominal is the closest I can get to contributing to all of this while still myself waking up and thinking about code every day.

• 31:43Speaker 01

  So let's kind of go down that because you mentioned the coding agents, the coding agent loop, and that's totally transforming software development at the moment, right? People are really trying to figure out what it looks like to be a software engineer, looking out a few weeks or months, or I think years is too hard at this point because stuff is changing so rapidly there. You had a public LinkedIn post about it's not yet doing that in hardware. AI is not changing hardware engineering to anywhere near the same degree. Can you expand on that? And maybe we can dig into some of the whys behind that and what it would take to actually accelerate or or expand it in that way?

• 32:19Speaker 02

  So one of our customers that I can't like give specific details about, but like they're trying to build a machine that's like the size of a building. And it's taking, you know, they're going as fast as possible. And it's frankly like the fastest I've seen any organization go on. It's still taking them months. But because of nominal, like this with their CEO, it says like, you know, we are looking at data and we are noticing problems. And we're catching them earlier. And so it's taken us 30 minutes to fix them instead of something catastrophic happening and us losing two days. So that is like the tightness of loop that is still like, that's an order of magnitude, right, that we're providing. But it is very different than like. I write some code and then I run a unit test and all of a sudden know like now the code is writing itself and running its own unit test, which is it's an order of magnitude, but like a microcosmic scale.

• 33:03Speaker 02

  I really want, I think as a, as humanity, we should like have the ambition of getting to that in the hardware world, but that requires this like concept of like, well, the hardware can be tested in a unit testable manner. And that feedback loop can then introduce the opportunity for something like a design agent to understand what's working and not working in a hardware design. And there, there's aspects of this, right? Like, I think there's amazing companies that are, you know, you can kind of give it certain parameters and it will like spit out like a proposed design and then there are simulations that you can run on it. But at the end of the day, you still have to like physically build something and wire it up. Maybe you're driving it out to a desert and flying it. And like those things just, just take a lot of time and human coordination. There are safety risks involved and I just think we're like

• 33:47Speaker 02

  willing to be pretty YOLO with OpenClaw, but like would not want to with a rocket fire or especially like any kind of flight that has a person inside the vehicle. And so to get there, we need to have some place for like the data to kind of be in, be aggregated and for people to kind of give thumbs ups and thumbs down on that data. Talking about the event catalog again, but if you have a sensor that's at a rocket test facility and you know, again, that rocket is only being fired for a few seconds at a time, a few times a week, the rest of the time it's sitting dormant. You don't want to drop any data points, but like those, you know, the eight seconds of tens of millions of dollars of investment, like that golden data asset, you have a lot of

• 34:25Speaker 02

  dead time in there that you can delete and filter out, understanding where those regions of interest and disinterest are, having the people who got the degrees in physics and mechanical engineering and are like working on top of a platform like nominal and telling us where the interesting data is, like what's anomalous, what's expected, what's anomalous, but fine versus anomalous, but dangerous, like these are the types of data assets that we like need to be accumulating if we ever hope to train kind of agents on the world of physical engineering.

• 34:51Speaker 01

  Yeah, I think what's interesting, one of the reasons that software is so amenable to agents is because you have that very tight feedback loop and you can give the agent the ability to just try shit and then see what happens and kind of iterate in feedback because they're probabilistic, they're not going to get it right every time, but if you can do that. So, looking at the hardware world, some of what I'm hearing is that feedback loop is harder because there's a human, you've got to print a thing and do a thing. The simulation solved this, can we get, or is simulation not good enough? Like, how do we get to that level of an agent can have a hypothesis, try a thing, get a feedback loop, and then iterate on that without a human having to be in that loop?

• 35:31Speaker 02

  This is where we're getting above my pay grade, but it's like, the simulation's not good enough and I almost go back to like, being like a curious college student and thinking about these things, but I start to wonder, like, is it actually better to send the agents thinking at the level of physics and simulation and building up from there versus trying to kind of like, give them simulation tools and have them jump in at the hardware design layer? Like, the quip I always share is that if you ask an LLM about the way a jet will perform, it's not thinking in terms of physics, it's kind of thinking in terms of humans who have translated physics into English and that it is itself composing the concepts of English summaries and kind of synthesizing some results to you, and then you have to

• 36:12Speaker 02

  go test if like, the physics end up matching that or not. And that is not good enough to like, actually make progress. Very different than software where like, you're, you're working in code, which they natively understand.

• 36:23Speaker 01

  So, a thing that I have observed is LLMs are systematically bad at a number of things, but one of those things is time. And you talk about, you're doing a lot of stuff with time and time series data. Is there an LLM component to nominal software, or do you cut that out and not deal with that at all right now?

• 36:39Speaker 02

  Oh, yeah, thank you for asking that, because I should be really clear. Like, we released just a few weeks ago, like, all of our customers have access to it now, but you can interact with a workbook agent, you can have a chat interface. And the reason we added that is because it actually reduces a lot of friction for our users, because we have to support so many complex calculations and visualizations inside our platform that, I don't know, it can take- A hundred clicks to build up a visualization of what happened during a flight, doing a flight test analysis. And if you have done this before inside our platform, you know, there are certain equations that like we can communicate about in English. And there's enough literature on the internet about what that means that the agent can get kind of right, understanding like, okay, well, speed is the square root of the sum of the squares of the component speeds.

• 37:22Speaker 02

  Like you can ask questions like this and it can kind of perform a lot of clicks for you. So that's something that we've built in right away and we think is really powerful. And then we honestly have a ton of stuff in our roadmap where we think there's more ways to reduce friction in our platform, leveraging these technologies, but then also like probably some exciting things we can do in terms of training on time series and using these architectures to help our users identify things that are interesting that even a large organization of a hundred smart engineers might miss. Lots of kind of like research that we're doing there and excited to see where it goes.

• 37:55Speaker 01

  Yeah, I mean, there's a lot of interesting kind of traditional ML that you could do in that space. And one of the often overlooked side effects of the boom of LLMs is we now have ML on APIs. Like you don't have to be an ML expert to start testing with and tinkering in training and all these things.

• 38:12Speaker 02

  Yeah, totally. I saw a really interesting talk one time that was just like reinforcement learning was like just getting going. And now everyone's been distracted by chatbots for a couple of years. I'm excited for when we go back to reinforcement learning. But it's similar to your question about simulation where like we really want our platform to be able to support, hey, this organization is plugging into an ML vendor or an ML consultant, they've hired some whatever. Or, you know, there's a feedback loop that you need help standing up where you might have the outputs of a test and want to be feeding that back into a model. The model's getting updated. That's going back to another team who's doing kind of like parameter design. At some point, we are going to have to kind of like vertically integrate there. I think that our customers tend to lead us in the direction of like asking for more technical expertise.

• 38:54Speaker 02

  But at the moment, it's like, you know, if you're training your own model, if you're bringing in something like Anthropic, whatever it is, like we want to kind of, as a platform player, be a good integrator with all of those.

• 39:04Speaker 01

  I guess that sort of leads to the question of like, where do you see this going for yourself for nominal? But also just sort of this broad space of hardware development and test over the next few years. And it's gotten almost cliche in software. I try not to project years anymore, but hardware is still slower. So I'm gonna, I'm gonna push you to, to project out a few years, not just a few months like we do in the LM world.

• 39:27Speaker 02

  I think we should be projecting years in the software world too. I think we shouldn't give ourselves that pass, but there's a lot on our roadmap and a lot that I see our customers doing. But one of the things that we are working with a few different partners on right now is how to be more efficient with testing. We have, I think in a way that's very natural, developed these regimes of like intensive certification and testing that are very expensive, both from a time and a dollar standpoint. And with the advances that we've been able to achieve in machine learning and building models of things like aircrafts, we can actually just do like, as you said, good old fashioned data science about like, hey, how do we achieve the same level of confidence or same level of kind of statistical inference with 100 data points

• 40:08Speaker 02

  instead of a thousand or a thousand instead of 10,000. We have some partners of the Air Force who dream of a pilot who's up in the air performing a test flight, being able to get feedback from something they did five seconds ago and informing what they're going to do.

• 40:23Speaker 01

  Oh, that's fascinating, right? You get to real time and you're like, hey, we saw a little bit of a flex on this. Can you try this maneuver and see what happens or something?

• 40:31Speaker 02

  Yeah. And if anyone knows what a kneeboard is, it's literally a clipboard that you have like almost duct taped to your thigh. That's like, these are the 10 things I should do while I'm up in the air experiencing G-forces. Like, it's very hard to do what these guys do. It's incredible. But how do we make that an iPad? And how do we make it so it's not predetermined, but dynamic? That's a dream. I think like the same kind of principles can kind of trickle down into like lots of aspects of hardware development. We see like vendor purchaser collaboration potentially happening, right? Like, how do you reduce redundant testing or increase the fidelity of the way information is communicated about what tests were performed and what the results were? As someone who's like worked in enterprise data, like I love this stuff. I think it's incredible. Some people maybe find it boring, but if you boil

• 41:14Speaker 02

  it down to like kind of the software problem, it comes down to these things like asset hierarchy and catalog and data lineage that we touched on earlier, which are just like so, so crucial to these organizations functioning well. And then it's this increasing drive to integrate the different roles in the design, prototype, development, initial manufacturer tests, and then like scale manufacturing. I think a lot Of these aspects of hardware development are going to be changing over the next couple of years. Like the way you lay out a manufacturing facility, it might be done by AI, maybe. Like that's an area where I know some companies are spending time. And so as those things speed up, is that shifting bottlenecks to other parts of our hardware organization?

• 41:56Speaker 02

  Or are you changing some of the paradigms? And for us, we just like, we bet that like being like sharing a single kind of data substrate across all of that is going to make everyone move faster. A really easy example to talk about is like, it used to be that a hundred people were focused on one test asset. And if you look at the way companies like SpaceX operate, like they think in terms of how do we get it? So one engineer can monitor a hundred assets, kind of inverting that, getting much more efficient. But we all know from the world of software, like when you give people that level of kind of like ownership, they actually develop more understanding and they can come up with innovative changes that would otherwise be lost in bureaucracy.

• 42:32Speaker 01

  So far, we've pretty much entirely talked about what I might talk about, like mega scale hardware, right? Like rockets and jets and all of these things. Are you working with like smaller scale stuff, drones or Internet of Things type stuff? Or other types of physical devices out in the world?

• 42:50Speaker 02

  I would say that not micro scale things. We really think that our software product can be applied in manufacturing and operational contexts where the devices themselves get smaller. But the scale, the number of them gets much larger. I would say that there are drones that we work with that are, you know, not mega scale. And where a lot of the interesting part comes from, it's not that you're flying once or twice a day. It's you have hundreds of them that are flying like dozens of times a day and that data challenge gets to be quite fun.

• 43:19Speaker 01

  I was gonna say, how does that change the shape of the problem?

• 43:22Speaker 02

  It changes it a lot because large mega scale assets, if there's a single big problem, you might, you might pause operations and be like, we need to figure out what happened there before we fly tomorrow. Versus it becomes more of the cattle than the pets where it's like, okay, we had a thousand problems yesterday. What's the one that's the most important for us to kind of drill into? Like when I think about, you know, the future of something like drone delivery, it's probably, you know, it doesn't need to be. Hundred percent accurate in the way that something like that, whatever the new aircraft that Boeing is developing for commercial flight needs to be. But it needs to be, you know, ninety nine point nine percent accurate. And that's more of an economics problem than a pure engineering problem. We talk about fleet monitoring and fleet observability.

• 44:05Speaker 02

  And then it's an interesting problem because it translates from just the, not just development and tests, but also to kind of like operations and maintenance. And again, it's like the logic that is being defined by the engineer who's first testing something versus trying to figure out how to ramp up production. Versus like, hey, this satellite is orbiting the earth or this drone is doing dozens of deliveries a day. It does not make sense to have to re-encode that logic in three different tools. That's again, like part of why we're starting Nominals. We think that can all be like framed worked away into a single platform. We are really focused on testing right now, but excited to kind of get more into operations and manufacturing in the future.

• 44:40Speaker 01

  Well, and that does end up smoothing your pathway a lot, right? Instead of having to, as you get, end up tool soup. Okay, it's time to, to operationalize everything. New set of software, got to figure it out. Well, I think we're coming close to the end of our time here. Is there anything we have not talked about yet that you think would be important to discuss before we wrap?

• 44:58Speaker 02

  Yeah, I think one of the things that I find exciting right now at Nominals, like we have this organic user growth in some of our large organizations that, you know, again, I was designing nominal, I was not thinking about the seven person startup where the CEO was using the data. And I also wasn't thinking about environments where we don't even necessarily know the users, but they're kind of sending each other links of like, hey, I did this analysis in nominal, can you check it out? But we're at a kind of point in our, our growth where, you know, we're starting to have dozens of people log into our platform in a given week that we haven't met before. With that comes kind of like data scale challenges. And we're also kind of on the side, like I said, trying to research how AI can be incorporated into our platform. So

• 45:39Speaker 02

  I'd say across all of that, if you're interested in picking up your software knowledge and kind of like walking across the aisle to the world of physical engineering, come talk to me.

• 45:47Speaker 01

  Yeah, for sure. I think the AI integration is an interesting one because one of the things enabled by LLMs is what I call like intent-based UIs. Right? So traditionally, if you're setting up a data dashboard, you need to understand a lot of the nuts and bolts and you need to kind of drive through and say, okay, this is where I'm getting this data and I'm putting it here and I'm putting it there. And like, do a lot of imperative design of your data dashboard, right? Do this, then do this, feed this to here and go here. Whereas with an LLM, if you understand broadly the shape of the data, you can just say, hey, I want something that maps a bunch of these inputs to some useful dashboards that I can watch.

• 46:24Speaker 01

  And the AI can just do it for you. And so to your point, you can open your user spectrum to folks beyond the initial experts who know exactly how to wire A to B. And say like, just those people who know what data they care about, or even just what problem they care about, let the AI figure out what data is relevant for that.

• 46:43Speaker 02

  Yeah, if I can riff on that for a second, it's like, I'm so excited about our platform play for exactly the reasons you're saying, where there's a lot of kind of custom, inexpensive user interfaces that we wouldn't have been able to build in the past, but now our users can build it for themselves, right? So that opens this aperture. And then I think the challenge then, and this is a conversation I'm having with our customers, is like, well, but then when it comes to tagging, cataloging, any kind of data write back, something like the events that you're trying to like canonicalize into a valuable proprietary data set, like it needs to converge. And so how do you give people the tools or give, you know, if it's agents building the software, how do you give them the tools to allow for that kind of intent-based user interface? But then, I don't know what's the right term here, like the organization needs to kind of actually agree and there almost need to be laws about like, well, okay,

• 47:29Speaker 02

  how is the data managed in a way that's not just individual, but organizationally beneficial?

• 47:33Speaker 01

  Yeah, absolutely. It's an interesting time, right? Because with great power comes great responsibility, right? If everybody can vibe code whatever they want, then you're in trouble because you get a lot of garbage in there. But if you can use that in a way that lets you empower people, but doesn't let them screw up the underlying pieces, doesn't let them make a mess, if that makes sense, then yeah, there's, it's really, it's an interesting world we're getting into.

• 48:01Speaker 02

  Yeah, I think we're early in the journey of getting more eyeballs and more data within our customers. And I think that these AI technologies will just actually accelerate that. And that's just good for everyone.

• 48:12Speaker 01

  Now, really quickly on that topic, there's an interpretability challenge, too, right? Because it's very easy. What is the saying? Lies, damn lies in statistics, right? It's very easy to get data that tells a story that may not be true if you don't understand how it's interpreted. So I'm curious, yeah, as you think about those user interfaces, how do you make them interpretable by the end user?

• 48:35Speaker 02

  I mean, that's something we haven't touched on, but like, since the earliest days of the company, like my paranoia in designing our user interface was like, if you're decimating, you know, you have a multi-hour flight and you're trying to kind of plot it. And a lot of people to zoom in on regions of interest, how do you make sure that those like zoomed out plots don't tell a lie and don't like hide a data point that could be critical? Because it could just be a split second of behavior across a multi-hour test that is really problematic and indicative of something horrible happening. You know, our users, they lose sleep because they are like, what if the data has something, a story being told in it, and I just am not seeing it? And I, that's the thing that makes or breaks this asset or system that we're developing. So again, I think we, right now, at least,

• 49:18Speaker 02

  have the benefit of like, our users have a culture of being like, super disciplined. And as long as the tooling is not getting in the way of that, which, you know, again, we prioritize heavily at nominal, then they will do the right thing. I'm, again, not to make multi-year predictions, but like, I am a little bit worried about like, if culturally we become too dependent on, you know, automation, or if the UI is being developed by an agent, and there isn't that, like, attention to detail that we have built into the product, like, do you start to have compounding misunderstandings? Like, that would be terrible. So I hope we kind of like, cross those bridges in less critical realms first, you know, whatever it is, there should be some kind of outage in software world, and people will kind of like, talk about how they built better risk management around

• 50:01Speaker 02

  it. Like, let's learn those lessons in a less safety-critical space, and then bring them to kind of like aerospace and nuclear engineering.

• 50:08Speaker 01

  So you're saying you don't want your rocket designed by OpenClaw?

• 50:11Speaker 02

  Not yet, not yet. Let's start with video games, and then we can move on to physical systems.