3D assets & simulation at NVIDIA

What’s the current reality and practical implications of using 3D environments for simulation and synthetic data creation? In this episode, we cut right through the hype of the Metaverse, Multiverse, Omniverse, and all the “verses” to understand how 3D assets and tooling are actually helping AI developers develop industrial robots, autonomous vehicles, and more. Beau Perschall is at the center of these innovations in his work with NVIDIA, and there is no one better to help us explore the topic! Sponsors: Fastly https://fastly.com/ – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Beau Perschall – LinkedIn https://www.linkedin.com/in/beau-perschall-4a00121, X https://x.com/bperschallChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: NVIDIA’s Omniverse https://www.nvidia.com/en-us/omniverse/usd/Beau’s GTC 2023 session around how to build simulation-ready USD 3D assets https://www.nvidia.com/gtc/session-catalog/?search=S52401&tab.catalogallsessionstab=16566177511100015Kus#/Tech blog around Omniverse and SimReady assets https://developer.nvidia.com/blog/new-cloud-applications-simready-assets-and-tools-for-omniverse-developers-announced-at-gtc/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:42Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International, and I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist with Lockheed Martin. How are you doing, Chris?

• 0:57Chris Benson?

  Doing good, Daniel. How are you today?

• 0:59Daniel Whitenack?

  Oh, I'm doing great. I had a conversation at breakfast on Monday this week with a company from the UK doing drones and automated or autonomous drones. And I felt very prepared for that because you've talked to me so many times about aeronautics and drones and all that. So thanks for your prep.

• 1:19Chris Benson?

  No problem. Happy to do it, you know?

• 1:22Daniel Whitenack?

  Yeah, it was a good breakfast.

• 1:23Chris Benson?

  Just think of the universe of possibilities out there, you know, so many things.

• 1:27Daniel Whitenack?

  Exactly. Yeah. Well, speaking of the universe, or I guess rather the omniverse, or the metaverse, or whatever verse you want to think of, we're going to get into all the verses today.

• 1:42Chris Benson?

  We're going to be well-versed in those verses.

• 1:44Daniel Whitenack?

  Yes, we're going to be well-versed. Good stuff. We've got with us Beau Perschall, who is the Director of Omniverse Sim Data Ops at Nvidia, which I have to say is a really exciting title, one of the better ones we've had on the show. So welcome, Beau.

• 2:01Speaker 03

  Thank you very much. I'm pleased to be here. Yeah, I imagine that my title doesn't make a whole lot of sense to just about anybody. It's a lot of words.

• 2:11Daniel Whitenack?

  I bet it'll make more sense after this conversation.

• 2:15Speaker 03

  Hopefully so.

• 2:16Chris Benson?

  I was gonna say, you have a whole episode to explain it to us, so we're good.

• 2:21Speaker 03

  Fair enough.

• 2:22Daniel Whitenack?

  You know, I guess spinning off of kind of how Chris and I were starting that it would be awesome to hear about, you know, what does Omniverse mean? And also maybe a little bit about like your, your background and how you came to be working on Omniverse. So this intersection of what I understand, some type of 3D stuff and AI and simulation. What was that journey like and how can we understand generally what Omniverse is?

• 2:50Speaker 03

  Sure. So Omniverse is Nvidia software. It is our computing platform for building and operating metaverse applications. And again, it's not necessarily so theoretical. These are like industrial metaverses. These are, you know, whether you're designing and manufacturing goods or you're simulating your factory of the future or building a digital twin of the planet, which Nvidia is doing to, you know, accelerate climate research. Omniverse is a development platform to help with that kind of simulation work and it's doing it in 3B.

• 3:30Daniel Whitenack?

  Yeah. So it's not just those people without the legs kind of hopping around in a place.

• 3:37Speaker 03

  No, this is very practical. As a matter of fact, we have big and small customers that are using it. Over 200,000 downloads for Omniverse is a platform that you can get from the Nvidia site. You've got companies like BMW that are using it to plan their factory of the future and part of that is worker safety. So they have to have legs. You know, you can't simulate the ergonomics of if you're doing a repetitive task, are you going to hurt somebody by doing it? Or are they in danger of getting, you know, hit by something in a work cell or something on the assembly line? So there's all sorts of simulation around that kind of information as part of Omniverse.

• 4:18Speaker 03

  It's a really broad platform. You know, it's designed to be extendable so that, you know, customers can come in and write their own tools and connectors. It's not supposed to be just its own endpoint. In other words, we have connectors which are basically bridges to other applications, whether you're coming from manufacturing side like Siemens, or you're coming from architectural software like Revit, or you're coming from animation software like Blender, or Houdini, or Maya, or Unreal for that matter. All of that data can be aggregated through USD. Universal Scene Description is the file format that Omniverse is based upon, which was

• 5:03Speaker 03

  a Pixar open file format. It is very robust. And basically, we figure we're the kind of the connective glue between all of these platforms so that simulations can be run inside of Omniverse, but all the data can move in and out. It's not like captive data. Hopefully, that gives you a little bit of a background of Omniverse in and of itself. It is a visual platform.

• 5:30Chris Benson?

  It does. That sounds fascinating. And as you know from our pre-chat, I know a little bit about Omniverse before coming into the conversation. But I know that there is a lot of confusion about like how this fits in with all the other, you know, we kind of, we were joking in the beginning about the various verses that people are hearing. There's a lot of lingo out there. And so, and as recently as yesterday, a friend of mine named Kevin texted me, and I haven't replied to him yet, but I will have by the time this is aired, texted me saying, "I don't understand this verse thing, and I know that you're involved in this. Can you explain it?" And I think Kevin represents a lot of people in that way.

• 6:12Chris Benson?

  And so, could you, we've heard multiverse, we've heard metaverse, we've heard now, we definitely have heard Omniverse, and how does all of, can you give us some context for how this whole industry fits together so that as we dive into Omniverse, back into it in just a moment, we kind of have a sense of where it fits within, you know, and some of the other companies. We know you're with Nvidia and you're doing this great work, but we've heard things from other big companies, you know, the usual array of social media and cloud companies. So can you kind of set the stage for us a bit on it?

• 6:45Speaker 03

  A bit, yes. A metaverse is a very... Loaded term and everybody has kind of their own connotation of what that is. For Nvidia, certainly, we consider Omniverse a tool, a platform to help enable an industrial metaverse, something that is real world, not only that is, that can do simulation, but can communicate with the real world and back. So there's this kind of bidirectional, you know, messaging that that's aspirational for us. That's where we want to be able to be. So that if you have a production line, you can actually understand what's the uptime of the equipment in there and then basically schedule maintenance or be able to do factory

• 7:30Speaker 03

  planning and optimization so that you're getting the most throughput you can at any given moment if you have to move materials around a facility.

• 7:40Chris Benson?

  Let me ask you a question there, just to draw out the distinction. As you just now were defining it, you kind of said industrial metaverse and I'd like, if you would, I know that people are reading things all the time and you know, like there's a more of a generic concept of metaverse and then obviously there are certain companies that were formerly known as Facebook that have kind of taken the word as a brand in some ways. I sensed that you were using the more generic version, obviously, of metaverse. Could you define what that is, what a metaverse is, so that we can kind of understand what the Omniverse branding of that fits into?

• 8:14Speaker 03

  Sure. So the metaverse, again, very overworked term, I think, but in general, it's the next evolution of the internet. Instead of having connected pages, you'll now have connected, living ecosystems, living kind of worlds, if you will, that actually can intercommunicate. You'll do hopping between those worlds as opposed to just moving between pages. So it's all based on kind of this 3D centric, you know, representation of our existence in some ways. You've seen it, you know, the gaming industry has things like Fortnite and Roblox already that are very much kind of persistent, ongoing worlds.

• 8:56Speaker 03

  The metaverse is designed to take that to a much broader level in everything from entertainment to business and industry. And so Nvidia is taking their

• 9:08Speaker 03

  Software platform and the hardware that supports it to help real world applications. I mean, it's why we're building an entire platform essentially around how we start to do weather prediction decades into the future called Earth Two, so that we can start to help with unlocking the climate as far as that goes. We have customers like Ericsson build digital twins of cities so that they could go ahead and place cell towers in optimal locations for maximum coverage before they ever deploy in the real world. So trying to find real world values, that's kind of the distinction between, you know,

• 9:48Speaker 03

  the gaming space and kind of the entertainment or personal spaces that the metaverse can represent with Meta and different companies that are helping work on that. Everyone thinks everyone's competing. That's like saying, who's building the internet?

• 10:02Daniel Whitenack?

  Fair enough.

• 10:03Speaker 03

  You know, at some level, it's like, it's going to require all of us cooperating at some level. There's so much greenfield as far as this space goes that, yeah, it's really exciting.

• 10:13Daniel Whitenack?

  Yeah, I really love this sort of parallel that you've given or metaphor of the internet because some of the applications that I've heard you talk about, like it's making some connections in my brain that's making this maybe a little bit more practical to me. So when I think of like the internet generally and what you can do on the internet and what has happened with with the internet over time, there have been things that happened in the real quote unquote real world that kind of had a parallel on the internet, right? Like I can go into a bookstore and I can buy a physical book. Well, now there's a way for me to do that on the internet. But then the internet also had this sort of segment of new things that didn't happen before the internet, but now happened because of the internet.

• 10:58Daniel Whitenack?

  Would you say it's similar in terms of what you're seeing with the metaverse space, these 3D worlds and the omniverse in terms of some of what you've talked about, like the cell tower thing, like in theory, you could do that in the real world and, you know, learn what you need to learn. There's probably cost, you know, advantages to not doing that and that sort of thing, but it's a parallel there. Is there another set of things, like, I don't know if this would fit into the climate modeling stuff or other things that you're talking about where you can do, you know, legitimately sort of

• 11:29Chris Benson?

  New types of things in this world that maybe we don't know the full extent yet, but we're beginning to see those. Do you see it that way? Or you're much more plugged in?

• 11:38Speaker 03

  Absolutely. I certainly see autonomous vehicles, which is another, you know, big industry for us with our DriveSim platform that's based on Omniverse is that if you're trying to simulate multiple kinds of traffic situations and different scenarios, a lot of them you can't capture in the real world. They're dangerous. You know, what you want to do is be able to train the algorithms to react accordingly before you ever get it on to the real world. But you also want to have the connectivity so that the way that it's handled is that it doesn't matter if the data coming in is synthetic. So the sensors,

• 12:18Speaker 03

  the LiDAR and radar on the car with hardware in the loop, essentially, you're now at that point of saying it can't distinguish whether it's a real world scenario or it's a simulation. It treats them both equal. So that sort of thing, I think, is absolutely, you know, critical to safety. I think that that also kind of gets to the industrial and the manufacturing side of things as well, is that there will be ways to train things in more efficient ways as well. So you're saving cost. You're training a robotic arm on a production line for a new task. Instead of having to take that work cell down in the real world, accrue costs while you're going through and programming it and testing it. Now you can go in and actually

• 13:03Speaker 03

  test it and teach it essentially in the simulation and then just pass all of that data back to the physical world so that the robot now just changes its program pretty much on the fly. That's a huge, huge kind of benefit.

• 13:18Chris Benson?

  For a moment, just as we kind of finish up, kind of the how, what the ecosystem looks like, and you're kind of talking about these use cases, I wanted to go back for one second and talk about with both Nvidia and the other organizations that are participating in this with their various solutions, some gaming, some not. What does that evolution of a User, if we're going into the future, a short distance, and it's becoming commonplace for users to have different destinations in terms of metaverse-style 3D worlds. In the beginning, are they all very distinct and separate, almost like using a separate application on your laptop, where you close one and you go into another one?

• 14:01Chris Benson?

  Or is there any, will it take a while to get to connection between those different types of environments? And what does that cross compatibility across multiple environments start to look like?

• 14:13Speaker 03

  I think that's part of why I was hired a year ago, was to help kind of solve this. I was hired to create a new standard that we call SimReady for 3D content specifically, because, yes, what you're describing is essentially a walled garden kind of approach, where everyone's doing their own thing, and nothing talks to one another, and it's all kind of disjointed. And that's not the goal of the metaverse. The whole idea of the metaverse is to be interconnected and allow people to move and allow data to move. And so with Omniverse being based on a file format called USD, again, Universal Scene Description, a very robust format,

• 14:56Speaker 03

  now what we're trying to do is understand how to standardize that, how to make it so that, based on your needs, and this is what's been fascinating for me in the last year, because I did not come from a data science background. I was a 3D artist for 20-plus years. In fact, I learned 3D before the internet was a thing, just to carbon date myself. I had manuals and didn't see my family for months, and had to work on super slow computers. But we're now getting to a point where interchange is absolutely paramount. So everyone is starting to look at it from a very cooperative place. So USD being an open file format, being something that is open source.

• 15:39Speaker 03

  We've got connections to the Academy Software Foundation, which helps try and manage standards, the Linux Foundation for Standards. It's a long, hard process to figure out what is valuable. For everybody, because as you can imagine, everybody's use cases is different. It's, you know, what BMW is trying to do is going to be different than what a watchmaker does, or what Ericsson is doing, or what autonomous vehicle manufacturers are trying to handle directly. And what we're trying to do with SimReady is build this framework that allows SimReady to have flexibility based on your needs.

• 16:22Speaker 03

  If you're doing synthetic data generation, where you need thousands and thousands of images to identify what a car is, that's one need. So you need semantic labeling. You need something in the data, in that 3D model, that says, "I am a car." Fairly simple, but you can get very specific, even within a single 3D model. These are the tires. These are the doors. This is the windshield. And you can start to semantically label more and more granularly based on your needs. I've been dealing for just under a year, trying to learn what is important. And it's like drinking from the fire hose. Everybody has different needs.

• 17:03Speaker 03

  Daniel, I assume, being a data scientist, that you have very specific needs for the kinds of data that you are, you know, processing. And how you want that data organized is somewhat different than, you know, an Nvidia researcher might need. So instead of trying to funnel people into one workflow, we're trying to make sure that SimReady becomes this living, breathing organism that must evolve over time and has that flexibility. So that, you know, we're providing the planter and the soil and saying, "Plant your tree. Here's how you do it," so that you can customize it to your own needs.

• 17:46Speaker 03

  Again, another practical example is with SimReady, specifically a piece of content.

• 17:53Speaker 03

  Right now has semantic labels. And what was shocking when I got here was finding out that our research scientists, I was like, well, what semantic labels are you using right now? What's your taxonomy? How are you identifying things? And what's coming with those data sets? And they're like, we get nothing. It's like, what? Yes, they are basically having to kind of like from whole cloth, create their own semantic label taxonomies. I'm like, well, that's crazy. But what taxonomy would you like to use? And they're like, well, everybody was kind of a little bit different. And so it's like, okay, what do we do there? So there's kind of the starting point. And in terms of a simple taxonomy, that will allow people to identify the car.

• 18:37Speaker 03

  But some people want to call it a car. Some want to call it an automobile. Some want to call it, if you're a French researcher, you might call it a voiture, if my high school French, if I remember it correctly. It's like, how do you kind of synchronize all those? And it's like, you're crazy if you try. So essentially, what we've done is we're building a framework and a reference implementation to be that planter. So that we can say, here's how you can implement it for your specific needs. And what data do you want to manage? Do you want physics? Do you want to have rigid body physics on the objects right now? Great, you can go ahead and add those. You know, we have that as part of PhysX that is built into the Omniverse platform. So when I said simulation, it can do collisions and collision detection.

• 19:22Speaker 03

  But there's more. When you think about building digital twins, you're trying to represent the real world as accurately as possible. And that is an endless quest, which is why it has to evolve over time. We'll build stuff now. But in the future, we'll have more sophisticated electromagnetic materials that have thermal properties and have sonic properties and have, you know, deformation, you know, tensile strength and things like that, that we'll want to build in so that the simulation can actually process it. So it is a You know, the rest of my life's work and then some, I think. You know, it's going to continually evolve. So what we're trying to do right now is in the very early days, set the standard up that

• 20:06Speaker 03

  it does have that ability to kind of breathe and move along as we get more sophisticated.

• 20:13Daniel Whitenack?

  Well, Beau, I love how you kind of brought what to me is honestly a little bit of an intimidating subject, which is this whole area of 3D. And, you know, I'm sure you have a different perspective, you know, coming from the art world, but I'm very much, let's just say, I shouldn't design any sort of applications that humans should, humans look at with their eyes. I'm not that kind, I don't have that skill. So it's a little bit intimidating for me to think about these spaces, but I think the practicality that you just described around, you know, I can definitely see even applications. I don't work in manufacturing, but I can see those. But even in my own space, you know, I work in natural language processing and language.

• 20:58Daniel Whitenack?

  And of course, a big area that is, is really neglected in the NLP space is sign language, which by its very nature is a 3D thing, right? A lot of people might think, oh, it's just like, you know, hands and you can look from one direction. Well, there's gestures, there's facial movement, there's 3D movement that happens with sign language. And, you know, if you want to, for example, have an avatar where you could, you know, type something in and the avatar, you know, signed in American Sign Language or Japanese Sign Language or something, that, that's a 3D environment and would require certain labels, right?

• 21:39Daniel Whitenack?

  Around, you know, facial features and hands and all of those things. So all of that really connects with me well. I'm wondering if you could kind of break down this SIM-ready project that you've been working on and maybe think about it from the perspective of, let's say, I am a manufacturer. I'm coming into the space. I want to kind of... Figure out, like you say, you've got the planners ready. What does it look like for me to come into the space and think about, you know, my use case and then map that onto Sim Ready, the standard and the file formats and the 3D space? What's sort of required for me to enter that space as it stands now?

• 22:22Speaker 03

  That's a great question because a lot of people understand is 3D is still very hard to achieve in any kind of degree of fidelity. And Omniverse is trying to help create the highest visual fidelity on top of simulation fidelity possible. So that kind of pyramid of what it takes to build 3D content in the first place is still difficult, even with photogrammetry and the new Nerf technologies and things that can help start to capture that. And those are going to evolve. And Nvidia, being an AI company, is certainly pushing into those areas to make it easier for kind of this art asset acquisition.

• 23:05Speaker 03

  But in terms of what it takes right now is, well, let me back up here. I'm kind of front-running myself in my head, is that essentially, with 3D being difficult, it's hard for kind of anyone to come in and just have a data set and be able to do a lot with it.

• 23:28Daniel Whitenack?

  I've never taken an animation class or anything. So you're working with that sort of clay.

• 23:34Speaker 03

  That's okay, neither have I. Right. You know, essentially, it's adding the value on top of the art asset. So if you're a manufacturer or if you're doing sign language, one, you have to have the asset library. And, you know, ML researchers and data scientists have a voracious appetite for content because you can't have just one thing to train against. It is thousands or tens of thousands for for humans, it's diversity, not, you know, just in terms of age and ethnicity and sex and, you know, clothing and look and facial I mean, it's endless there just to be able to train the model, you know, with as little

• 24:19Speaker 03

  bias as humanly possible. The same thing for any other kind of research where you're using 3D. I had a researcher ask me early on when I first started, can I get everything you find in a garage? I was like, no, that's an unbounded question. You we had, let's focus. What do you want? Is it, you know, are you?

• 24:39Daniel Whitenack?

  There's a lot of strange garages out there.

• 24:41Speaker 03

  Exactly. Am I a woodworker? Am I a mechanic? Am I a hoarder? Is it my garage? Who's all of that kind of comes into play is to kind of like focusing down on first, what is the data set consist of? And then what metadata is important for the use case? So that's really kind of where SimReddy starts to differentiate is it says, okay, now that I've got this data set, what adds the value to it from this set of tooling that we're building? Also on top of Omniverse, so that at the end of the day, I can take beautiful art assets, stuff that has no metadata for simulation or for AI at all, be able to push

• 25:22Speaker 03

  them through this tooling to add semantic labels, to add physics, to add physical materials, to add all of the kinds of things that matter to the, you know, dimensions of the object, whatever other kinds of metadata are important to that customer, and then be able to ex- validate it and export it so that now you've got a data set that a data scientist can consume directly, practically, without having to spend their life trying to figure out how they add the value on their own. So that's where, because at the end of the day, I don't think Nvidia envisions themselves or me having a team build all the content in the world for people.

• 26:02Speaker 03

  We want to enable all of the suppliers for BMW, the Siemens and Kukas and, you know, companies like that who build infrastructure and build content to also embrace the idea of SimReady and the tooling so that all of that content just plays nicely together. And then, again, it flows into and out of other simulation platforms. So if you're pushing it somewhere else, it's a USD file, so that data is available to you regardless of what platform you're using it within. So that's really kind of the benefit there.

• 26:36Chris Benson?

  I just would like to extend exactly what you just said. Could you give us, and we often will ask guests just to kind of give us a nice clarifying way, like what you just said is you described the concepts of going through that process. Could you give us either a fictional or a real world, whatever works for you, and I suspect you probably have one ready to go, of like, pick a manufacturer or whatever you want, and kind of walk us for a moment at a high level through the steps of what they're doing, where you reference Omniverse, you reference SimReady, you reference the things in context in a use case, so that we kind of follow your footsteps through that, and it kind of brings the concepts into a very tangible, you know, touchable kind of of understanding.

• 27:19Speaker 03

  Right. So we actually have a project ongoing right now, and I can't mention who, but essentially, there is a pick and place robotic arm on a conveyor system that actually has sensors to indicate where parts are on that platform at any given moment. And what they want to be able to do is build that simulation inside of Omniverse, so that both the simulation can drive and time the real world application, and the real world application can report back, so that there is this kind of cyclical nature of having data moving both ways. So it is, you know, a feeder drops, you know, a part onto

• 28:01Speaker 03

  the conveyor belt, the system always knows where it is, it can count it, it can track where it is on the, in the process. When the arm is supposed to pick it up, it knows how to do that and move it into the right location. Those are the kinds of use cases where now if you have sim-ready content that knows, it can identify itself, this is a package, this is a conveyor, this is, you know, this part of Omniverse can trigger when the real light sensor is tripped and be able to understand that as a, hey, this is where this product should be. So if the simulation or the real world is off, they can adjust on the fly. So that now you've got kind of this self-fueling, um, round-trip ability to track content that way.

• 28:46Chris Benson?

  So is it fair to say like you would take assets, 3D assets, and you would apply, uh, USD, the universal scene descriptor, to it to give it the context so that it is, quote unquote, "sim-ready", and you can use the sim-ready tools on those assets to do whatever it is you're doing?

• 29:07Speaker 03

  Right. USD is actually the file format. I mean, but it's more than that. So it, and most applications now either export USD directly, just like you would if you're working in a CAD application, you might export a DWG file, or a DXF file, or something like that, or a SolidWorks part file. If you're in manufacturing, you can now export USD directly in many of these apps. They're all starting to get on board, which is great for the 3D industry. Cause I can tell you that when I was coming up, every 3D app, every tool had its own 3D file format. And so nothing played well together. It was always a nightmare to try and get content from one place to another,

• 29:51Speaker 03

  without question. It wasn't like 2D imagery where a pixel is a pixel. 3D is much more complex as far as that goes, orders of magnitude. And so now that we all have, we're starting to kind of hone in on USD as a primary file format. We have Technically, there's another file format that's open that's run by the Kronos group called GLTF. And it is essentially a web standard for 3D. And I was part of the group that was helping kind of define the standard for 3D commerce. So that you could see things on your Apple phone and spin them around in the websites and things like that as well. So that's kind of the JPEG version of the 3D.

• 30:32Speaker 03

  Well, the USD file is more like a layered Photoshop file, much more robust. But they play very well together. And Omniverse supports both of them too. So this is great.

• 30:43Daniel Whitenack?

  So one of the things that you mentioned briefly, Beau, which I think is really a fascinating topic, but also a really important topic for the future of sort of practical artificial intelligence, machine learning, is the idea of simulated data. Now, you kind of briefly mentioned this topic of, you know, creating 3D worlds, all the file formats and the things that are needed to label those to make them useful for data scientists. You talked about the example of kind of digital twin running in parallel with the real world robot arm. Could you set the context now for usage of this technology for synthetic data production?

• 31:28Daniel Whitenack?

  And from your perspective, where you've seen people do that, maybe a couple of examples successfully? And maybe help people understand what synthetic data means and why it might be useful?

• 31:40Speaker 03

  Sure. So synthetic data, as far as I have been involved, is essentially generating randomized, we call domain randomization, taking lots of objects, randomly placing them in scenes with all of their labels in place, so that you can train machine learning for computer vision, to be able to identify, you know, something in a room or in a space or in an environment. So it doesn't matter what the lighting conditions are. It doesn't matter what the material is. It doesn't matter the orientation of the model. It could be upside down in some arbitrary orientation.

• 32:24Speaker 03

  But at the end of the day, when you have that image or that sequence, video sequences or whatever that does all of this, the computer algorithm can always pick out whatever that piece is. We have a version of our CEO Jensen and we call him Toy Jensen and his little 3D model, toy model. You've probably seen him in our GTC talks and keynotes and they wanted to do kind of a Where's Waldo for SDG for him as well, just to be able to train. Where is he in a scene with all sorts of other random 3D content? And so that you would change lighting, you would change materials, you would change the orientations of everything to train the algorithm to be able to spot Toy Jensen, no matter where he was in the

• 33:08Speaker 03

  scene, how much he was obscured by blocks or sofas or things like that. From a more practical standpoint, think about what furniture manufacturers are trying to do today with augmented reality. You know, they want to be able to scan your room. They want to eventually say, I know that that's a sofa and that's a chair and that's a table and I want to be able to replace it with my stuff instead and show you what my stuff looks like in your space. And so having that computer vision trained against a huge variety of content now gives their algorithms the ability to to kind of find and identify that stuff with

• 33:52Speaker 03

  high accuracy or, you know, good fidelity.

• 33:55Chris Benson?

  I just wanted to say tongue in cheek that I think finding Jensen is not as hard as you say, because he always has his trademark motorcycle jacket on. I'm just saying, it's like, it's always the jeans and the motorcycle jacket. So.

• 34:07Speaker 03

  He does indeed. They actually put him in the midst of all of our marbles content, the real time sequence that they put together for a real time demo for GTC two years ago. And there's hundreds of elements. And so he could would get pretty obscured where you couldn't see either his jeans or his jacket.

• 34:24Chris Benson?

  Okay, fair enough.

• 34:26Speaker 03

  And you would see like a part of his gray hair and that would be about it.

• 34:29Chris Benson?

  Gotcha.

• 34:29Speaker 03

  Some fascinating stuff. You know, from what I'm trying to do with AI, just to kind of circle this all back around to Sim Ready. Is that AI is important for SimReady in the future, too. I mean, again, just starting, less than a year in. But my vision is to work with our data researchers as well. So that at the end of the day, instead of having a tool that you manually have to process content with, why wouldn't our SimReady tools live in the cloud as a service for people to upload their content? And doesn't matter how materials are named. Is it named metal? Is it named wood? Ideally, AI would help us identify what that material should be to name it properly.

• 35:12Speaker 03

  And then do semantic labeling on it and be able to apply the right physics. I mean, so that you could upload your library. No one has to get involved. The system now can process your library, give you a dashboard and your data set that is now valuable. That's my long term vision specifically for AI for SimReady.

• 35:30Chris Benson?

  I'd like to ask you, is, and part of this just comes from the kind of the, you know, I work for a company that has to deal with edge scenarios that are adversarial and challenging in all sorts of ways. And so I'm always going to that. One of the things I'm always curious about is, is we look at simulation and, you know, built on the larger cloud approach that we've been doing for the last 20 years, 15 years, I guess now. As you move these capabilities and you're talking about having 3D assets, you're doing augmented reality and you want to be able to merge those, as you mentioned, like with the room with your own stuff. But there's an infinite number of variations there.

• 36:06Speaker 03

  Absolutely.

• 36:07Chris Benson?

  That we could talk about from a use case standpoint. As you get out and you're doing things that are away from the cloud, you either don't have enough bandwidth to get all the GPU computation, you know, from the cloud back to where you are out in Everest Base Camp, because the, you know, that actually probably does have enough of an internet connection. But let's say you're up in Camp Two and you're doing something in a fairly remote region. How do you envision these starting to merge into that in terms of being able to have a consequential user experience, you know, something that's impactful? In terms of augmented reality where you're combining all

• 36:46Chris Benson?

  With these 3D assets that are SIM ready and it's merging with your world when you don't have bandwidth and cloud assets immediately available due to technical limitations and the How is Nvidia thinking about because I know you want it to be everywhere. So how are you thinking about Bringing this future that we're all hurtling toward and that you're inventing into those spaces that are not just I'm on a gigantic internet connection, sitting in my office, doing my thing.

• 37:13Speaker 03

  Right. I mean, certainly Nvidia wants things to live in the cloud. As much as any company at this point, and it's been, you know, Jensen publicly announced that in the keynote for GTC this past fall. And having kind of that unique position of having hardware and software With our GPUs and the Omniverse platform give us some distinct advantages where you can actually Do quite a bit from your own small workstation in terms of streaming content and how we might do that in the future. Honestly, I don't know. You know, to be completely fair, I don't know what that looks like at this point, you know, I'm only a year old here. So.

• 37:54Chris Benson?

  I would argue that Nvidia is very well positioned for answering that question because You're not strictly 100% in the cloud. You there's a I have bought products from you that I can go place into a computer That is not in the cloud or that may have a connection, but I'm doing the GPUs out on the edge. You have a whole, a large product line Of things. So I do think that you're well positioned for that, but I think it's a fair answer to say I don't know because we're moving fast and you know, what's the cliche?

• 38:21Speaker 03

  It is still early days. There's no question and it there's going to be a lot of evolution. I know that, you know, what we're focusing on this year As a company is awe inspiring and I can't wait to see how we progress throughout the next 12 months or 11 months now. To get closer to those goals. So it's. There is a lot to be done, but yeah, I don't know.

• 38:43Daniel Whitenack?

  As you do look to the future of your own work and what Nvidia is doing, but maybe also like, you know, now that you're in this space of 3D and interfacing with data scientists, thinking about how that can influence AI, how AI could help you build the things that you're doing. You know, what's on your mind as you're looking towards the future? What excites you? What sorts of opportunities really keep you up at night and really keep you thinking about, you know, the potential in this space? I know that, you know, you mentioned your background in art. And of course, this last year has been an amazing year in terms of the generative capabilities of AI. And, you know, that even sparks things in my mind about how the things you're working on in 3D, you know,

• 39:27Daniel Whitenack?

  interface with that sort of generative capability. What are you thinking about? What are you looking forward to as you're moving forward?

• 39:33Speaker 03

  For me, one, there's almost nothing to not be excited about, including generative AI. But for me, it's when it comes to SimReady and kind of my focus is really the sophistication of what we're trying to achieve with AI. It's starting to kind of understand the, you know, what the value is today and how you start to extend it forward so that we can start to extrapolate out much further forward. Building that bidirectional communication between the simulated world and the real world. Wow, cannot wait to see how that starts to really kind of manifest where you have data cleanly flowing both ways

• 40:15Speaker 03

  and things start to synchronize so that you're not just simulating at this point. You are now kind of replicating things that way. I think that's huge and I'm, you know, I was lucky enough to be around when 3D first kind of went mainstream where you could have computer PCs, consumer PCs instead of, you know, $50,000 workstations that could do 3D. You know, and with AI, I feel like we're kind of in that similar early phase of creation and understanding so that there is just this enormous green field in front of us to kind of explore. And it's going to take all of us, too. It's not just NVIDIA. I want to make that clear. It's like, we're focusing on things that we feel we have distinct advantages on, but

• 40:59Speaker 03

  we need collaborators. Again, it's back to the adage of, how do you build the internet? With a lot of people, a lot of cooperation. There's so much opportunity across the board that we've all got to kind of pull together and do it.

• 41:11Daniel Whitenack?

  Awesome. Well, I think that's a super inspiring and encouraging way to close things out. It's been an awesome conversation, Beau. Really appreciate you taking time to talk about all the things that NVIDIA is doing in this space and the things that you're working on around standardization and making things useful and practical for people like myself and Chris. And yeah, thank you so much for your work and your contributions.

• 41:33Speaker 03

  Thank you, guys, for having me. This has been a blast. I've enjoyed it thoroughly.

• 41:46Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now, if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changetalk podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.