959: TypeScript on the GPU with TypeGPU creator Iwo Plaza

Scott and CJ sit down live at JSNation NYC with Iwo Plaza, creator of TypeGPU, to dig into how WebGPU is unlocking a new wave of graphics and compute power on the web. They chat about shader authoring in TypeScript, the future of GPU-powered AI in the browser, and what it takes to build a killer developer-friendly graphics library. Show Notes 00:00 Welcome to Syntax! 00:32 What is TypeGPU? High-level overview and why it exists 01:20 WebGPU vs WebGL – the new era of GPU access on the web 01:47 Why shader languages are hard + making them accessible 02:24 Iwo’s background in C++, OpenGL, and discovering JS 03:06 Sharing graphics work on the web vs native platforms 03:29 WebGPU frustrations that inspired TypeGPU 04:17 Making GPU–CPU data exchange easier with Zod-like schemas 05:01 Writing shaders in JavaScript + the unified type system 05:38 How the “use_gpu” directive works under the hood 06:05 Building a compiler that turns TypeScript into shader code 07:00 Type inference, primitives, structs, and TypeScript magic 08:21 Leveraging existing tooling via Unplugin + bundler integration 09:15 How TypeGPU extracts ASTs and generates TinyEST metadata 10:10 Runtime shader generation vs build-time macros 11:07 How the AST is traversed + maintaining transparency in output 11:43 Example projects like Jelly Shader and community reception 12:05 Brought to you by Sentry.io 12:30 Does TypeGPU replace 3JS? How it fits the existing ecosystem 13:20 Low-level control vs high-level abstractions 14:04 Upcoming Three.js integration – plugging TypeGPU into materials compute shaders 15:34 Making GPU development more approachable 16:26 Docs, examples, and the philosophy behind TypeGPU documentation 17:03 Building features by building examples first 18:13 Using examples as a test suite + how docs shape API design 19:00 Docs as a forcing function for intuitive APIs 20:21 GPU for AI – browser inference and future abstractions 21:11 How AI examples inform new libraries (noise, inference, etc.) 21:57 Keeping the core package small and flexible 22:44 Building “TypeGPU AI”-style extensions without bloating the core 23:07 The cost of AI examples and building everything from scratch 23:41 Standard library design and future of the ecosystem 24:04 Closing thoughts from Iwo – OSS, GPU renaissance, and encouragement 24:34 Sick Picks & Shameless Plugs Sick Picks Iwo: Perogies Shameless Plugs Iwo: Syntax Podcast Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott: X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Scott Tolinski?

  Welcome to Syntax. Today, we're going to be talking to Evo Plaza about TypeGPU and all of the amazing things you can do with it. So we're here at JS Nation, myself, CJ, welcome CJ.

• 0:14Speaker 01

  What's up?

• 0:15Scott Tolinski?

  And Evo, we found him, just got offstage here, so we figured we would chat about TypeGPU.

• 0:21Speaker 02

  Thank you so much for having me. Yeah, this is amazing. Longtime watcher of the podcast.

• 0:25Speaker 01

  Ah, he mentioned he roots for me in CSS battles.

• 0:29Scott Tolinski?

  Oh, yeah.

• 0:29Speaker 01

  He's a CJ bot in the comments.

• 0:31Scott Tolinski?

  CJ bot in the comments, man. Yeah, so let's start with TypeGPU, which is something that came onto my radar recently. I made a video about it. And can you just give the audience a little bit of high level, what is TypeGPU, how it came about?

• 0:46Speaker 02

  Well, I feel like we're in this new kind of era on the web. Like, for a while, all we could do was WebGL and graphics. And FreeJS is the main library player. But I feel like now we have a very good opportunity to try a different direction, try something new for TypeScript enthusiasts. And that's what TypeGPU is. TypeGPU is a wrapper around WebGPU, which is an upcoming web standard, which is almost supported in every browser. Linux is lagging behind, but hopefully next year. And it gives you very low-level access to your GPU while still having that TypeScript developer experience. And our main frontier

• 1:30Speaker 02

  right now is allowing you to write shaders in JavaScript, which normally what you have to do is you have to write them in a special language called WebGPU Shading Language, which might not be super accessible for existing JavaScript developers. So we want to just make that easier.

• 1:46Scott Tolinski?

  Yeah, I found the shader language to be very tough to pick up, especially for like, somebody who's used to programming in JavaScript, you're always thinking about looping in regards to executing this, then this, then this. But with shaders, you're executing tasks on every single pixel at once when you're doing graphics? Is that still the case with Type? GPU and how you yeah man that is a weird way to like shift your brain.

• 2:16Speaker 02

  Yes.

• 2:17Scott Tolinski?

  Have you worked a lot with web GPU before in embarking on this or graphics programming in general?

• 2:24Speaker 02

  Yeah, I mean, my first experience with programming was was C++ and OpenGL a while back.

• 2:30Scott Tolinski?

  Yeah.

• 2:30Speaker 02

  Um, and I only got into jazz because they didn't they didn't have um C++ compilers on my school computer.

• 2:39Speaker 02

  So now I'm here now I'm writing into us.

• 2:42Scott Tolinski?

  Have you ever built a visualizer for uh what is it uh?

• 2:46Speaker 01

  Like Winamp?

• 2:46Scott Tolinski?

  Yeah, like a Winamp visualizer, yes.

• 2:48Speaker 02

  Oh, I wish. It would have been amazing mostly like game game development with with OpenGL and and C++. And I mean, that was there's always amazing, but I feel like JavaScript is a is a really amazing platform and the browser in general for for creating experiences that you can just share very quickly with people. I was always frustrated that, you know, I was building on my local machine, but if I wanted to share it with a friend that has a Mac, I I couldn't do it. I would have to buy a Mac. Um, and that just wasn't in my budget with on my zero dollar budget. So I feel like the web opens that up a lot, and I'm thankful that I get to use this experience um in at work and working open source full time.

• 3:30Speaker 02

  I've been working a little bit with web GPU before, and type GPU is mainly the result of my frustration with web GPU. Because it's it's super powerful, but it's very low level. If you want to communicate with um with web GPU, it's essentially like you're trying to talk to a server, but the only way to talk to it is to send instructions to it in a different language and uh send a binary to it. You can't send like even numeric values like you have to package them up as a float free to array and send it over. If you want to do more complex structures, you have to think about padding in memory,

• 4:10Speaker 02

  how the GPU is going to interpret that, what the padding rules are, and if you get that wrong, well, you get a wrong pixel and good luck trying to debug that. So that was our first frontier, just making that. That connection between the shader and JS easier and we do that through ZOD like schemas.

• 4:25Scott Tolinski?

  Yeah.

• 4:26Speaker 02

  Um, so we were very inspired by by libraries like ZOD, like valid bot archetype. Where you define the schema once and you get both the type safety and the runtime checks. And we do a similar thing, but instead of runtime checking and validating the object. We use it to serialize and deserialize plain old JavaScript objects, arrays and primitives. And what that allows us to do is you have a single source of truth that generates both the Wexel shader code. And allows you to send and download arbitrary data from the GPU. And that was our first frontier. We finished that up about a year ago. And since then, we've been pushing that forward to allowing you to write shaders in JavaScript.

• 5:07Speaker 02

  Where now, thanks to that integration, you can write your shader and refer to something that lives on the CPU. And then you just have that connection, both at runtime and on the type level. And that was very, very important for us. And I feel like it unlocks a lot of possibilities similarly to how TRPC and Svelte kit allow you to run your function on the server without having to worry about REST APIs and stuff like that. You can now just call a shader function without having to set all of that glue up yourself.

• 5:39Speaker 01

  Can we talk a bit about how that actually works? Cause I mean, so like if you, if you, first of all, go check out the docs. The, the really good docs.

• 5:45Scott Tolinski?

  Really good.

• 5:46Speaker 01

  But you're writing code that just looks like TypeScript and I don't know if I've seen a library like this before where you're writing TypeScript code, but there's this little use GPU directive.

• 5:55Speaker 02

  Yeah.

• 5:55Speaker 01

  And at the end of the day, your code actually gets run through a compiler, like you wrote a compiler that takes your code that looks like TypeScript and actually turns it into shader code or like web GPU code.

• 6:06Speaker 02

  Yes.

• 6:06Speaker 01

  So yeah, can you talk a bit about how that works? And so in your talk, great talk, by the way.

• 6:11Speaker 02

  Oh, thanks so much.

• 6:11Speaker 01

  You had an analogy of like front end versus back end. There's typically like this translation layer of like TRPC or even like Svelte kit with like remote functions. And you mentioned that this type GPU library is kind of like that for the CPU and the GPU. It's like, it makes it much easier to. Write code that communicates between the two. Can you talk a bit about that and how this all works?

• 6:32Speaker 02

  Yeah, yeah, of course. I mean, the, I feel like the serialization and the type safety is a bit easier to, to explain and start from. The, the code structure, I mean, how, how you would approach it normally is just a lot of TypeScript generic magic.

• 6:50Speaker 02

  But essentially, if you, like, we have a couple of built-in primitives that we just export and we know the types of them. And if you use them in our APIs, we're able to extract the information about what you're trying to use. So it's essentially equivalent to having like multiple overloads over a function and we choose behavior based on that. But the more, more complex the structures, like structures that define shapes of objects or, or arrays. Yeah, they're just basically a bit more complex in terms of the TypeScript types. So we just go over the fields that you defined in the schema and we save that on the type level. So that when you refer to that same struct, we're able to properly change up the API and react to it. I mean, like TypeScript is Turing complete on the type level.

• 7:36Speaker 02

  So you can technically do anything, even run Doom, of course. But yeah, it, like, as far as TypeScript magic goes, between defining type annotations and Doom, we're still like pretty close to here, not doing super advanced stuff in the, on the type inference, uh, on the type inference front. But yeah, that's essentially how it works. And afterwards, uh, I mean, since you can just access those objects on in JS, if you write your shaders in JS, you also have access to that. And the types naturally, naturally flow from your CPU code to your GPU code since you can import stuff from other JS files and the types propagate naturally thanks to it being in the same language.

• 8:22Speaker 02

  So we don't even have to like write our own custom language server because it's, since it's just TypeScript and it's semantically correct TypeScript. We can leverage existing tooling and hook into your existing bundler, uh, thanks to unplug in the on JS.

• 8:38Speaker 02

  Ecosystem, thank you to Daniel Rowe.

• 8:42Speaker 02

  We're able to create a single transformation plugin that looks at your JavaScript and tweaks it a bit so that it works. I can talk about it later if you guys want. But we're able to target pretty much every bundler on the planet, thanks to the work of the Andrew S. Team and Unplug-in. I don't know if you guys are familiar with the project. But yeah, it's an amazing piece of tech. And if you guys think that you want to build something, first look at the Andrew S. Website. Nice. You're most likely going to find it there. It's cool. It's amazing. But yeah, we basically look at your JavaScript source code. We look for functions that are marked with the use GPU directive.

• 9:25Speaker 02

  And we attach metadata to it so that at runtime, we can associate a function with the metadata that we can use for generating the shader code. So the actual build step is quite simple to showcase and to explain. At least visually what the code actually turns into. You could think that we replace the JavaScript functions with the shader code, but that's not actually what happens. To keep it flexible and to allow the shaders to react to the user's settings or your settings as the developer, or as I showcased in my talk, being able to accept callbacks from the user to inject into your shader code. We do have to defer shader code generation to runtime.

• 10:10Speaker 02

  You can do that at build time if you use another Unplug-in gem or Andrew S. gem, which is called Unplug-in Macros. I really recommend that as well. It's like Bun Macros, but for anything. But what we do is we extract the abstract syntax tree of the function that you marked with use GPU. We turn it into a very compressed format that we call Tiniest, which stands for Tiny E S D, like Tiny Embedable Syntax Tree. But we just, we figured out that name just for the sake of reserving that name on N P M.

• 10:43Speaker 02

  So we inject that metadata into the user's JS bundle. And then at runtime, whenever somebody gives us their function, us, I mean, the type GPU API. Whenever somebody says, hey, type GPU generate a shader for this function. We're able to see, okay, did you associate metadata with this function? Okay, cool. Let me extract that metadata. And then based on that, we generate the shader code. The details of like the how we traverse the tree and generate Wizzle from it. It's going to be in my talk, hopefully on YouTube someday soon. It's definitely like a complex process, but thankfully, like at the end of the day,

• 11:24Speaker 02

  the code that you write is pretty similar to the code that ends up on the other end. And we really wanted to make sure that we don't abstract away too much of what's happening on the underlying platform. The only thing that really changes is the language that you're writing it in, which to us was really powerful because we can build examples like the Jelly Slider built by Connor Adretzko.

• 11:45Scott Tolinski?

  The Jelly Slider is really good, like killer demo.

• 11:48Speaker 02

  It blew up in a day. We were really surprised to see it, but the reception's been amazing. And the only negative comments that we saw was like, now my manager is going to make me implement this into my website.

• 12:03Speaker 01

  It has 3D all over the web now.

• 12:05Scott Tolinski?

  And if you want to see all of the errors in your application, you'll want to check out Sentry at sentry.io/syntax. You don't want a production application out there that, well, you have no visibility into in case something is blowing up and you might not even know it. So head on over to sentry.io/syntax. Again, we've been using this tool for a long time and it totally rules. All right.

• 12:30Scott Tolinski?

  Yeah, I guess, speaking of 3D, one of the comments that I got on my video when I talked about type GPU was wondering about how does this replace something like 3JS? Or how does this fit into that ecosystem? Because I think a lot of people's introduction to any kind of graphics programming on the web for web developers might have been through React 3 Fiber or 3JS or anything like that. So do you want to maybe touch on how does this fit into that picture?

• 12:58Speaker 02

  Yeah, of course. I mean, to us, a type GPU was always meant to be a very succinct and small wrapper on top of a GPU. And therefore, we don't abstract. I mean, it's good that we don't abstract away too much, but also the users have to implement a lot from scratch. If you look at the Jelly Slider example, there's a lot of code that goes into it because we have to implement the rendering logic. It does ray marching, it does reflections and shadows. And because we write that ourselves, we are able to control every aspect of it and optimize around it, which is great. But for people that want to have just plop a 3D model on their

• 13:40Speaker 02

  website, FreeJS is the way to go. I'm definitely not recommending like you to build everything from scratch. But for those that want to delve deeper into optimizing and have that low-level control over what's actually happening, what code is running on the GPU, I can really recommend starting off from type GPU and learning that ecosystem. But I feel like type GPU still has a place alongside FreeJS. We definitely don't want to like replace and build a new Free JS from scratch. Free JS is an amazing ecosystem and has a lot of community support and resources

• 14:16Speaker 02

  out there. So what we're hoping is that the low-level building blocks built by libraries or by hobbyists that specialize in graphics programming can build them using type GPU and then hook them into Free JS. We have an upcoming Free JS integration. Hopefully, by the time this interview is online, we would have released the 0.9 version of type GPU. Yeah. And with it, we have a Free JS integration where Cool. you can replace any part of your material with a type GPU function. Cool. So let's say the like the lighting logic and everything is still handled by Free JS,

• 14:58Speaker 02

  but you just want to change the color of the material. You can just call from or to TSL, which is the node setup that Free JS has. Pass in a type GPU function and you have a TSL node that you can plug into your material. And the same goes for compute shaders. If you want a way to write compute shaders in a more JavaScripty way where you have if statements, for loops and everything, you can do that. You can do from, you can do two TSL, pass in a type GPU function, dot compute. And you're going to have a compute shader working and interfacing with 3JS. So that's, that's really exciting. It's, it's been in the works for a while and I'm really happy to see it come together.

• 15:39Speaker 02

  And hopefully it's gonna like, uh, make people more, hmm, interested or, or have a feeling that they can do more with 3JS now and delve deeper into those details and play around with them.

• 15:55Scott Tolinski?

  Yeah. Yeah, I, I, I think type GPU will go a long ways to, uh, reducing just that, like, ah, man, every single time I've tried to learn web GPU, there is just such a tremendous amount of dread that comes over you when you start to work in it. You're like, oh, this is way, I'm way out of my depth in this. So like, yeah, hopefully that reduces that. Because yeah, 3JS, things like that, React 3 Fiber, they're so approachable, but, um, they're good introductions to wanting to get, get deeper with that. One thing that I really admire about your project is the documentation. The docs are really great. There's a lot of awesome examples. It's very readable and parseable and you can get in, get your hands dirty for me.

• 16:39Scott Tolinski?

  I'm like the type of person that really loves to get in there and tweak values and see what happens. What was like the strategy there in developing docs? Was that always a thought or is it just something that you personally like in documentation?

• 16:53Speaker 02

  I, I, I mean, I feel like I've, I've always liked writing about stuff, but, um, it's definitely something that required a kind of a, a shift of a mindset. A little bit of background on type GPU. I mean, when, when it started, it did have a use case. I was building it for my master thesis, which involved, um, doing super resolution of a game running in real time on the local user's machine. And I wanted to do it in the browser to just have a better time working with the GPU. And WebGP was just coming up and I was building up this API. So when type GPU, at least in its first form, before we were able to assemble a bigger team, it did have a use case of connecting different libraries together.

• 17:36Speaker 02

  But once we started working on it, we didn't really have like a project or a product that we could use type GPU in. So there was no like driving force to add APIs to type GPU that would help with a specific use case. So we were kind of needing, we were, we kind of needed to create our own product to have a use case for type GPU. And examples kind of became that. We were just like, okay, well, I want to do this using type GPU. So let's build this. What is missing? Okay, let's like plug in these APIs. Let's create them. Okay, does this work fine? Okay, let's create another example. No, it doesn't generalize that much. Let's try to think it through.

• 18:18Speaker 02

  And I feel like, like we, we build a lot of these examples before we ship our stabilized features in type GPU, because we want to ensure that like it's, it's both fun to use and it, it works for, for those particular use cases. So that kind of came naturally where, I mean, we, when we shipped type GPU at first, like we had a dozen examples or something right from the start, because we wanted to make sure that these APIs worked. And now they're kind of, I mean, they're cool to, to peruse around. But also when we introduce some breaking changes, they're great end to end tests, right? We want to ensure that we don't break anything. So we always run these examples when we change something and see if anything breaks.

• 19:00Speaker 02

  And as for the docs, I mean, we really wanted to explain how it works at first, but we kind of noticed that whenever we were describing a new API, when we were having a hard time explaining what it did, or it wasn't intuitive to explain, it was probably a wrong API.

• 19:17Speaker 02

  So it was very beneficial to try to explain it to both ourselves, the rest of the team,

• 19:26Speaker 02

  and to see if it's explainable. In like a paragraph or or if it's if it like if it's easy to learn step by step. Um, and it led us to a lot of exploration of of other things that maybe are easier to explain. And yeah, that's that's kind of the driving force. I feel like docs are usually seen as this. Um, last step of like, oh.

• 19:48Scott Tolinski?

  Yeah, I gotta do the doc.

• 19:49Speaker 02

  Yeah, I I I wrote this and it works amazing and now I have to write about it.

• 19:53Scott Tolinski?

  Yeah.

• 19:54Speaker 02

  Um, and I it takes time away from me actually building it, but. Um, I feel like in our case, when we did this mind shift of I I want to build this cool thing. So I have to explain it to my teammates and. Uh, like figure out what APIs I need, then it kind of comes naturally. And, uh. Yeah, we're still pushing forward with that, uh. Philosophy, I guess. Uh, so expect more examples, more examples to come.

• 20:21Scott Tolinski?

  Uh, a lot of people talk about GPU, web GPU in general, being not just for graphics, but for interfacing with the GPU to do things like AI in browser. Is that a focus of type GPU at all? Or is it just something that it enables?

• 20:37Speaker 02

  Yeah, it it definitely enables that. I mean, we, uh, we're definitely trying to build out the ecosystem ourselves. If we need it for, say, an example, our main approach for like building use case specific stuff for type GPU. Um, is usually we create a couple of examples that might have this functionality and we see what's shared between them. Um, and that happened for the noise library that we have, uh, which, uh, implements a lot of the basic like random functions and and patterns that you can plop into your materials or or, uh, uh, shaders. And with AI, we have one example that does this. We have, uh, uh, minist digit inference.

• 21:19Speaker 02

  So we can draw a digit and it's, uh, figures out what that digit is, uh, on your local GPU. Um, and I feel like once we, once we build out more examples that are that are using AI, um, we're gonna see an abstraction emerge there that we can extract into a separate library.

• 21:35Scott Tolinski?

  Yeah.

• 21:35Speaker 02

  Um, but we really want to make sure that type GPU itself doesn't, um. Maybe not get tainted, but like doesn't involve itself with the use cases that it can be used for. We really want it to be open for interpretation, I guess. Yeah, yeah, yeah. So whenever we do build some use case specific stuff, we release it as a separate library. And at least to me, it's, uh, it might be useful for people wanting to, wanting to create their own libraries that hook into this system. Where, uh, they can look at the simpler libraries that build on top of type GPU and use it as a template for, for their own like, uh, code, or even their own library if they want to create one.

• 22:14Speaker 02

  So it, it both ensures that our API of the core library is flexible enough so that we can build cool libraries on top of it. And not have to wait for like a feature request for, hey, can you expose this API for us? Cause we need to hook into that. By, yeah, by restraining ourselves to only put the essentials and low level stuff in the core library. And having to use that limited API set, we expose that same API to the end user. And I feel like that's, that's very powerful for customizability and extendability. So yeah, answering the question directly, we are working on AI stuff, but every AI example is very involved, usually involves, um,

• 22:54Speaker 02

  a lot of research around which model is best to use, or even training our own model. And coding the specific layers that need to be a part of it since, uh, at least right now we have to build all this stuff from scratch. So it's taking a while, but once we have a couple of these examples, we'll definitely look at, um, giving people a better abstraction. Call.

• 23:15Speaker 01

  That makes sense. It's, it's cool to see how you've architected the library because you import from standard library. And like you've kind of separated these things out to their purpose use. Uh, so it makes sense that maybe if there's enough of a use case, there might be like, uh, type GPU AI or something like, but it's, it's not included in the main package, but you can, you can grab it if you need it.

• 23:34Speaker 02

  Yeah.

• 23:35Speaker 01

  So we're getting pretty close to the end. Um, is there anything we haven't asked you that you want to say? Or anything you'd like to tell the world?

• 23:43Speaker 02

  If there's anything I'd like to tell the world, there's so much.

• 23:49Speaker 02

  Follow the Syntax Podcast.

• 23:51Speaker 01

  Oh, appreciate it. Thank you.

• 23:52Speaker 02

  Yeah, I listen to it every time I'm on my way to work. It's an amazing way to catch up on all the newest JS ecosystem news. And yeah, I'm really excited to get to work in the open source. I encourage everyone to try out TypeGPU and to contribute. To look at the docs and see if it's something that you'd like to dabble into. I feel like graphics and GPU stuff is having kind of a resurgence on the web.

• 24:23Speaker 02

  And it's exciting to be part of that wave and to hopefully make it better. And make the ecosystem a bit more approachable.

• 24:32Scott Tolinski?

  Yeah, thank you for doing that, making it much more accessible for people like me, for sure.

• 24:36Speaker 01

  Absolutely. Do you want to do Sick Picks?

• 24:38Scott Tolinski?

  Yeah, let's do Sick Picks.

• 24:39Speaker 02

  Cool.

• 24:39Scott Tolinski?

  Okay. Yeah, so Sick Picks, for people who don't know, is the part of the show where we talk about things that we enjoy and like. And Eva, we've kind of put you on the spot here.

• 24:47Speaker 02

  I know.

• 24:48Scott Tolinski?

  Just ushered you up here.

• 24:49Speaker 02

  I should have known.

• 24:50Scott Tolinski?

  So is there anything that you're really just enjoying in life? Could be anything.

• 24:56Speaker 02

  Pierogies are great.

• 24:57Scott Tolinski?

  Oh, yeah, yes.

• 24:59Speaker 02

  If anybody's not into Polish food yet, try it and you will be. And come to Krakow. Krakow has an amazing tech space emerging.

• 25:12Speaker 02

  And if you come, let's hang out, let's talk graphics stuff.

• 25:15Scott Tolinski?

  Nice. Awesome. Yeah.

• 25:17Speaker 01

  Cool. So thank you so much for being on. Lastly, a big shout out to JS Nation. Thank you for giving us the space and letting us use this to do this interview. This was great.

• 25:24Scott Tolinski?

  Yeah, it's been awesome. Thank you so much.