936: Realtime LED Wall With React + Websockets (I Let Strangers Control It)

Scott, Wes, and CJ dive into Wes’s Hackweek project: a real-time, web-controlled LED grid. They break down the hardware build, custom 3D-printed diffuser, ESP32 microcontroller, and Cloudflare durable objects powering live pixel art, GitHub activity displays, and interactive web drawings. Show Notes 00:00 Welcome to Syntax! 02:03 Wes’ Hackweek project: a web-controlled LED grid 03:52 The hardware: LED panels, soldering, and power WS2812 LED panels 06:38 ESP32 microcontroller and WLED firmware explained ESP32 Microcontroller 10:57 Power supply challenges and injection fixes 15:05 Debugging and testing a DIY LED matrix 15:56 Shorts, blown circuits, and melted wires 17:58 Designing and 3D-printing the diffuser for crisp pixels 21:29 The software: Cloudflare Durable Objects + Party Server Cloudflare Durable Objects Party.server 22:18 Real-time sync and state management across clients Party Client 28:43 Connecting the server to the LED hardware 41:51 Open access fun: scripts, NSFW images, and moderation Cloudflare tunnel 44:10 Live demos 45:34 Future plans: stats, rooms, and making it always-on Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott: X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Wes Bos?

  Welcome to Syntax. Today, I'm going to be detailing my Hack Week project, which is my realtime web-controlled LED grid that I put together. So Sentry has this thing called Hack Week where you get a week to build whatever it is that you want. Maybe a product. Scott and CJ did some really cool stuff. I opted to do something that was a mix of both hardware and software, as well as letting the wider internet be able to control it. And it was a super fun project. I learned a lot about it. And today, we're here to talk about all of the different pieces and get into the nitty gritty. There's a lot of really fun stuff and a lot of fun stuff. You're going to learn about it. I should say, this is going to be a pretty video-heavy episode.

• 0:44Wes Bos?

  If you're listening to this on the audio podcast, keep it going. You're going to learn a few things. We're talking about durable objects and web sockets and React and microcontrollers and all that. But if you want to cut to the video version of it, either on Spotify or on YouTube, I'm going to recommend that since there's going to be a lot of visual stuff.

• 1:04Scott Tolinski?

  Hell, yeah.

• 1:05Wes Bos?

  What's up, Scott and CJ?

• 1:06Speaker 02

  Not much.

• 1:08Scott Tolinski?

  Not much. I brought a salad to the party. So yeah, that's what's up. That's what's up. No, no, no, it's taken out. I'm just excited to see how you built this. I love stuff like this. I was very jealous when you started doing this project because I would love to build a little drawing interface like this. So no, it's super cool. And I gotta say, just you always seem to bring a level to Hack Week that is something that makes me personally want to beat you and step up and do something better. So shout out to you for always being so ambitious about Hack Week because I'm a competitive type of person. And I feel like if our team didn't go hard on the Hack Week, then it would be less fun.

• 1:49Speaker 02

  Yeah, and I'm excited to learn how you did this because it is, in a sense, a full-stack project because you've got hardware, you've got a front end, you've got a server that's got to sync these things. It's realtime. Yeah, I'm excited to hear all the parts and pieces that make this thing work.

• 2:03Wes Bos?

  So I'm gonna show you what I built here and then we're gonna break down all of the different pieces. So there is obviously the hardware LED grid here. There's a diffuser panel that has been 3D printed over top of it as well as with the frame. Then there's a web app where you can draw on it. If Scott and CJ, if you guys go to local.wesbos.com, you'll be able to go ahead and draw on. You can upload. One of the features is you can drag and drop images. You can obviously draw on it in real time. I do have a couple utilities so I can paint my GitHub contributions. Give it a sec. It's going off to the GitHub

• 2:43Wes Bos?

  API and pulling down all of my activity. Boom. So that's my GitHub activity in in real time being painted to the LED. I will note that this is very washed out in the video. In person, this looks amazing. It's like so crispy, but so like clean cut pixels. And I'll explain how I got that in just a sec as well. But man, it looks awesome. I've been, I'm very happy with it.

• 3:15Scott Tolinski?

  That's so exciting.

• 3:16Wes Bos?

  A couple of other things you can do with it. I hope to like put some social stats on it or be able to put like maybe build status of the syntax site, maybe some stats for the podcast or sentry errors, account of how many open errors we have on our sentry. There's just like a lot you can do with this. Now that I have this like sort of base thing built up, I am now able to just like put whatever it is that I want on there. Okay, so let's get on into how it works. Let's start with the hardware and then we'll get into the actual software side of things. So first of all, the the LED grid itself

• 3:59Wes Bos?

  is 2,000, 2,300 and something pixels. So what it is, is I bought these little 16 by 16. They're about, I don't know, 10 bucks, eight bucks a piece. And it's 256 different individually addressable LEDs. And how that works is that you send power to all of these LEDs, but then you can also send data signal to every single LED. And you can change it to be any color that you want, right? They're RGB LEDs. Then you can actually chain them together and make an entire matrix of them.

• 4:42Wes Bos?

  So what I did is I took nine of them and I put them into obviously a three by three grid and soldered them all together. I actually ended up undesoldering all of the leads on the back of them and just going, doing my own soldering because these clips were just a little bit too, too big to fit behind the frame that I made.

• 5:03Scott Tolinski?

  How long did that soldering take you? Cause I'm so slow.

• 5:06Wes Bos?

  I'm getting better at it. It's just that I had to, I did it like several times. Because my initial, the initial time that I did it, I, I used some like pre-made frame that somebody designed and I couldn't get access to the back. So I kept breaking it, trying to like, put the wires underneath it. It was so annoying. So I ended up just scrapping the entire thing that I found online and designing my own frame for it. But I don't know, soldering maybe, maybe like an hour or so, just having to quickly desolder everything. And then I, I put leads on it directly to it.

• 5:40Scott Tolinski?

  And when you said you send messages to that, like what are you able? It's, it's more than just an on off kind of thing? Or like what, how intricate are those messages?

• 5:50Wes Bos?

  The LED grid itself, it obviously it has like a positive, negative. It's a five volt LED. There's also 12 volts that you can get, which I kind of wish I had done because this is, this is a lot of LEDs. It actually takes quite a bit of power to turn all of these on. And then what's powering it and sending the actual data, which is the third line on these is something called. And ESP32. And this is just a $5 microcontroller, which is absolutely wild. And it has a Wi-Fi chip in it, has a Bluetooth chip in it. And then it has all of these different pins. And these pins are used for sending serial data.

• 6:33Wes Bos?

  Most of them are used for sending serial data out through the data line. And on this little ESP32 runs a little piece of software called WLED. So WLED is just this really nice piece of software that will allow you to, you can run all kinds of different animations. You can change the colors of them. I use this in several places. The little on air sign that I have behind me, I have it off right now because it's blowing it out. The gym lighting, I'm currently doing my house in these LEDs. And WLED powers all of that because it's a piece of software that can then send the data signal from the ESP32.

• 7:18Wes Bos?

  It spits it out in serial and is able to deliver it to the different types of chips that run on these, these very common LED strips or LED grids.

• 7:27Speaker 02

  So would you say WLED is like a custom firmware for ESP32? Or is it something that's also running on top of the firmware? Like as a separate app or something like that?

• 7:37Wes Bos?

  No, it's a totally custom firmware that you flash directly to these chips. And it's, it's written in C, C++. I, when I did my controlling my Roomba, I dipped into that as well. Writing my own lower level code that would send serial data out of these different pins. But it's, it's quite amazing because WLED also has, like it has, has Wi-Fi. You can, you can open it up on any browser. You can get apps that connect to it. But it also has several APIs that will accept incoming data. And those are the APIs that I used in order to send data from my application to it. It's, it's running, it's literally running a web server on here.

• 8:23Wes Bos?

  Like this, that WLED is not running on my computer. It's literally running on any, it's not, not this one, but one that's behind here right now. Since I have so many LEDs on here, I actually hit the limit of.

• 8:38Wes Bos?

  The number of pixels that you can send over a data line. Because the way that the data line works is that it snakes through it. So you start the data at one pixel and it goes in and then into the next one, into the next one. And that's actually how it does it with power as well. It sends the five volts in and out. But once you start to, like, you can do a little demo with one of these really quickly. You don't even have to have an external power supply. But once you get into putting a lot of them together, you hit the limitation of how many LEDs you can possibly send data through, which is 2048. So I was slightly over the actual limit. So what I ended up doing is I split them into three different sections. So one, two, three.

• 9:23Wes Bos?

  And I have three different pins on my GPIO pins on the microcontroller. And those are each sending three different pieces of data out. And then you can go into the software and set up the layout of it. You can see here I've got all nine of the panels. And then it takes care of, as far as like you, the user is concerned, it's just now, it's just a one single grid. And WLED figures out how to properly send the data to each of them and in which order.

• 9:54Scott Tolinski?

  Is that a common solution to handle that?

• 9:59Wes Bos?

  Yeah, yeah, like to use WLED or?

• 10:02Scott Tolinski?

  No, to break up, to send it in groups like that.

• 10:06Wes Bos?

  The way that serial data works is that it, it sends high and low voltages. So if you're running like a five volt system, it might send like a, like a 4.5 or a five volt to show like a one, right? And then it might show, send like a, like a one volt or a 0.5 volt to show a zero, right? And you're basically just sending ones and zeros. And then, as you know, like voltages over long runs can, can start to drift and it can start to, can start to die out. So if you have a very long run, or if you have a, this is not a long run, but it is a lot of pixels, then the data can start to get weak as it goes through. And there's, there's solutions to boosting it.

• 10:47Wes Bos?

  But the, the better solution is to simply just chop it up into multiple segments. And then that way you can also send data out the different pins at the same time.

• 10:58Speaker 02

  It's a single controller that is sending the data across three. Because I guess you could have also maybe had three different controllers, but this lets you use a single one.

• 11:06Wes Bos?

  Yeah.

• 11:07Speaker 02

  And then split up the the signal. I guess you'll probably talk about it, but like this requires a lot of power too, right? So each panel is

• 11:13Wes Bos?

  Yes.

• 11:13Speaker 02

  Five watts. So like, is there a separate power supply for the the panel grid itself? And then it's just getting its data signal separately?

• 11:22Wes Bos?

  Yes, exactly. So

• 11:24Speaker 02

  Okay.

• 11:24Wes Bos?

  The ESP32 will run on on 5 volts or 3.3 volts. And then you can also buy little adapters for them as well. I'm a big fan of this one, where you can put like a 12 volts into it as well. And it will just, it'll has a buck converter, it'll it'll dump it down. But you can power like one panel off of a USB, like a one amp off of USB. But once you go over one amp, it's actually funny, macOS X will be like, we disconnected your USB thing because it was using too much power.

• 11:56Scott Tolinski?

  Yeah.

• 11:56Wes Bos?

  So once you, once you pull more than that, this, all of these together can pull 94 amps, which is 94 times what it should do over USB. I didn't do that because you you literally cannot buy a 94 amp 5 volt power supply. The the real solution, when you do bigger stuff like this, is instead of having to do more amps, what you do is you just increase the voltage. So I kind of regret buying 5 volt panels because it has required me to, in in absence of volts, I'm just pushing more amps. And when when people do like, like big things.

• 12:38Scott Tolinski?

  Stadium things?

• 12:39Wes Bos?

  Yeah, like a stadium, they'll use like 48 volts. Like, that's commercial, right? Or or 24 volts. And and and most hobbyists will use either 12 or 5 volts. So I almost do everything in in 12 volts now, just because it's you have to fuss a lot less with it. But I had bought these, I don't know, six months ago. And I just, I didn't think about that. But I had, I bought an external power supply, which is like, like this big. And then I also go into my W LED settings. And I explicitly set the maximum power supply current to 18 amps right here. And, and that is plenty bright. I don't even have it.

• 13:22Wes Bos?

  I have it at half brightness right now and it's, it is plenty bright. If any, any brighter than that, you gotta start worrying about like thermal problems, right? Like it already gets kinda hot.

• 13:35Scott Tolinski?

  Yeah. If you cranked it, what would happen? Would it catch on fire or would they explode like in the goofy movie?

• 13:41Wes Bos?

  If you cranked it, then you have to start worrying about cooling. And then the other thing about it is, is like heat comes from resistance. So what was happening with, with these ones is if the power has to go through every single LED in order to get to the 2300th LED, it was getting really, really hot. So what I did is there's the second wire in the middle right here. This is the power injection and the data has to go from pixel zero to pixel 2300. But the power can go anywhere. You can inject power at the end, in the middle, at the start, ideally as often as you can.

• 14:23Wes Bos?

  So what I did is I just, I desoldered it and I just ran power from the power supply directly to every single panel and that brought the, they were getting pretty hot just regularly. And then they, they cooled down right a lot after that because there was not as much. You're not trying to pull a lot of amps through this tiny little flexible circuit board. You're, you're, you're bringing it right to the panel that needs it. Yeah. Lot of fun.

• 14:48Scott Tolinski?

  This is so crazy. Yeah. My, my problems could have been solved with code. And like to me, it puts me in such a place of like, wow. I remember any time of hardware, hardware project I've ever tried to do, you know, you get things soldered up, you plug it in and then like nothing happens. And then I'm just like, okay, I guess I'm done with this. Like, how do you, how do you get better at debugging this stuff? You just understand how to check values better, whether that's a voltage or?

• 15:15Wes Bos?

  Like you can add logging. Yeah. You, you check voltage as well. Like, um, like the way I knew I needed is I, you, you check the voltage on the input. And then you check the voltage on the output. And then if you see a certain percentage of voltage drop, like if it's, if it's like 20% less, that's, that's not good, right? You should, you shouldn't see that much voltage drop over it. So there's kind of a lot. To CJ just uploaded some stuff to it. You kind of learn over time and then there's just things you you understand about the basics of of electronics and how data is is sent over the wire.

• 15:52Scott Tolinski?

  A lot of a trial and error.

• 15:55Wes Bos?

  Yeah, a lot of trial and error.

• 15:57Speaker 02

  Well, did you burn any circuits or bust any pixels during this project?

• 16:01Wes Bos?

  Surprisingly not.

• 16:02Scott Tolinski?

  Anything busted. You bust in.

• 16:05Speaker 02

  Did you release the magic smoke?

• 16:07Wes Bos?

  I.

• 16:07Scott Tolinski?

  Were you busting?

• 16:09Wes Bos?

  I have burned several of these out over the years, but I did not have a single anything in this case.

• 16:19Speaker 02

  Nice.

• 16:20Wes Bos?

  There was once where I accidentally crossed the positive and negative wires. And I turned it on, I plugged it in and it wasn't working. And then I immediately put my hand on it and I realized it, the wires were getting very hot. And I I silly didn't put a fuse in it at first. I had the fuse on my desk and I was like, I'll put that in at the end. And I luckily I grabbed it and unplugged it and realized I had crossed some wires, but I had created a short circuit and I didn't have a fuse in it. So it was just running the amperage over and over through the things and it was melting the wires, which would have caught on fire if I had left it.

• 17:05Wes Bos?

  So that was my one scary moment there and I, after that, I quickly put a fuse is like, like 10 cents, I should have put that in at the very first.

• 17:14Scott Tolinski?

  Well, that's the kind of thing that I would do because I just wouldn't know any better, you know?

• 17:18Wes Bos?

  Yeah.

• 17:18Scott Tolinski?

  Not put a few, yeah. And the next thing you know, you're catching something on fire.

• 17:23Wes Bos?

  I once, when I was working on the ATV at my cottage before I really understood how LEDs worked, the LED to to show like if you're in drive or reverse, they weren't working. So I tried to put the battery voltage straight to the LED and it it literally exploded and it LEDs are glass.

• 17:45Scott Tolinski?

  Yeah, that's what I'm saying. I want some.

• 17:46Wes Bos?

  Luckily didn't get in my eye. And the the solution is LEDs, like those little tiny little pin ones, those are 1.2 volts and I put 12 volts to it.

• 17:56Speaker 02

  Nice.

• 17:56Wes Bos?

  Too much. The diffuser itself, so the LED grid is is kind of hard to look at. You look like a gamer that just bought some LED strips and put them up. So in order to make them look nice, you have to diffuse them. And what I ended up doing is at first I downloaded a couple of things on 3D printer world and whatever, and printed them out, but I just was not happy with any of them. So I designed my own diffuser. And after a couple of trials, this is what I ended up with is it is a 0.3 millimeters of white filament.

• 18:34Scott Tolinski?

  Yeah.

• 18:35Wes Bos?

  And then I did black like baffles or grid on the end. And what that did is it gave me like a very crisp look. I did it all in white first and it bled a little, which is a cool, it's a vibe. But it wasn't what I was going for with the whole pixel art here. I also didn't want like very defined lines as well. So I printed it. I'm not sure if you can see this or not. Let me see if I can feel like a makeup reviewer. I printed it.

• 19:03Wes Bos?

  A couple of millimeters, like one millimeter up in up the side in white. And then that allowed me to not get nice, big, black lines. And then I did the rest of it in just black. And that was the that was the winning ticket to do it. And I had nine of them and ended up, I ended up just gluing it. I I wanted so badly to like make some sort of click together system. But like, I was doing way too much already with this.

• 19:31Scott Tolinski?

  Yeah, yeah.

• 19:33Scott Tolinski?

  You're saying your your diffuser's got a BBL?

• 19:36Wes Bos?

  Yes.

• 19:39Speaker 02

  Well, I started saying, this is one of my favorite parts of the build because like I've tried to mess with pixel arrays before, and I've never come across a good diffusing solution. So I love the fact that like each pixel is basically a black. Square. It has like, it has walls around it to prevent it from bleeding. And then the top layer is just thin enough so that it can shine through. Like, cause I don't have a 3D printer and like, when I was trying to do this, I was like, trying to use like foam board and pieces of paper. Or like, you could buy like craft paper that's like slightly opaque. But then you run into the bleeding issue. So like, it just looks blurry. But the fact that you have, like, you've isolated each pixel to then shine through, like, is awesome.

• 20:19Wes Bos?

  Yeah, it's, it's really nice to be able to, to do that. A lot of people, I do regret, or not, I don't, don't like that there are little lines in here. I modeled it with a 0.2 millimeter lip on two of the edges. And what that allowed me to do is to overlap them very slightly. So you didn't have a very defined, very defined line. But there still is a bit of a slight line. I think if, you could also buy like, um, like a halogen, you know, you go to like your dentist office and they have those like tube lights. You could buy that and I think I might try that at one point where you just print the grid. And then you put a single sheet of diffuser material.

• 21:01Scott Tolinski?

  Oh, yeah, yeah, yeah. Yeah, that's how my, it's actually funny. I mean, how my softbox works, although the grid is on top of the diffuser. Does diffuser and then grid, but that's for a softbox. That's a completely different application, but.

• 21:14Wes Bos?

  Oh, it's the same idea though. So, yeah, that might even work. I didn't even think about that to flip it around. But that, that was the hardware aspect of it. So that was, was relatively fun to play with, play with all of that. And then on the, the software side of things, there was kind of two pieces. There was the entire client side app, which is the real time drawing. And then there was the whole server side part, which is keeping track of what has changed, sending that data over to the LED thing. So I'll start with the server side, cause I feel like this is the most interesting. So I built a, something with CloudFlare durable objects.

• 21:57Wes Bos?

  And, and PartyKit, it's called Party Server now. The whole PartyKit thing doesn't look like it's being super maintained, but they are moving.

• 22:05Scott Tolinski?

  Really?

• 22:06Wes Bos?

  Yeah, but they're, they're moving it to something called Party Server, which is a bit more of a peeled back layer. Which, which I moved it in, in like an hour, cause I initially built it in PartyKit. So CloudFlare durable objects are essentially a like a JavaScript object or a JavaScript class. Hey, let me open up some code here.

• 22:29Scott Tolinski?

  You're going to remove the photos, the dogs we're putting on your?

• 22:32Wes Bos?

  Oh.

• 22:34Wes Bos?

  Oh, my goodness. Look at this, Scott. CloudFlare durable object solves the problem of how do you maintain state or how do you maintain like memory with a serverless function or or a CloudFlare worker where these things spin up and down when they're needed and they spin up multiple versions of them when when you need lots of them. And it's really cool because you can simply just stick data on the object and then that data is is available anytime you come back to that instance of the durable object, which is very similar to if you were to like just stick something in

• 23:15Wes Bos?

  memory on a long running server, except that when you restarted it, it's it's it's actually saved to a database. So it stays there. So the durable object was really cool. And then on top of that, Party Server gives you the ability to do like a real time. What that does is when somebody is is drawing on this grid, what's happening is that there is a WebSocket connection from your browser to the CloudFlare durable object server. It's a CloudFlare worker. It's running locally on my machine, but you could deploy this thing to the cloud as well.

• 23:56Wes Bos?

  And then every single time there is like, like, somebody has drawn to a one of these grids right here, it's sending that data over a WebSocket to the backend. And then the backend is then saving that grid state to the durable object. And then it is then broadcasting those changes to everybody else who currently has this application open. There is ability to even do rooms in this one as well. I have it built into the app, but when you come to the homepage, it's just the the default room. And it's really cool because like there's you can get into like doing database schemas and whatnot, but I'm just simply just, I just have a like a like a JavaScript map and set.

• 24:42Wes Bos?

  For maintaining all of this data, right? It's just an array of 2,300 pixels and that array tells you which color every single pixel is. And then that that data source of truth is is the Cloudflare durable object. And then anytime that somebody draws, that data comes into the server, it updates it and then broadcasts that out to everybody else that is on it. And in fact, as I'm drawing on this thing, it's actually not painted to this grid until it sends it from the client to the server. The server then updates it and then it broadcasts it back to everybody, including yourself.

• 25:27Wes Bos?

  So it doesn't paint it until it comes back from the server. So it's like the opposite of Scott's. It's not local first at all.

• 25:37Scott Tolinski?

  This type of tech, is this primarily used for stuff like this? You wouldn't go building your whole data structure of your entire application in something. So would you or would you?

• 25:49Wes Bos?

  No, no, you you would stick your your regular application in like a D1 database or something on Cloudflare. The idea with the durable object is that it can maintain and share, I'm calling it memory, but just like just the state of the object. Imagine you had a JavaScript object that had a property called name on it and you set that to Scott, right? And then you come back in six months, that JavaScript object will still be there. And then you can also share that object with with other people that have it. So that is awesome for any time that you want to have long running processes or come

• 26:33Wes Bos?

  back to something, because the problem with with serverless compute is that it spins down. And then if you want to save literally anything, or you want to share literally anything across multiple instances of that serverless function.

• 26:49Wes Bos?

  Then you have to end up sticking that in a database somewhere or stick it in key value just so that you can share it amongst anyone. And this is simply just like variables that exist and can be shared for everybody.

• 27:03Speaker 02

  I guess on top of that, the client side library is what gives you the sync. Because I guess you could use this as like persistent data for your serverless functions, but being able to broadcast that state gives you the real timeness. And that's more of like the WebSocket client. Is that accurate to say?

• 27:19Wes Bos?

  Exactly. Yeah, that's so. There is something on the client side. There's something called party client. And then they also are working on something called party sync, which Scott, you'll probably like to hear as well, which is just like, yeah, all of this this data as well. But the party client is what connects the web sockets from the client to the actual durable object web sockets.

• 27:45Scott Tolinski?

  Is partykit a part of Cloudflare now?

• 27:48Wes Bos?

  Yes.

• 27:48Scott Tolinski?

  Okay.

• 27:49Wes Bos?

  Partykit was made by Sunil Pai, and he worked at Cloudflare for a long time, left, created partykit, and then that was acquired. So he's back there.

• 28:00Speaker 02

  Wow.

• 28:00Wes Bos?

  And he's working on all kinds of interesting agent stuff as well. I shouldn't have said like partykit is not no longer maintained, but they Cloudflare, basically, they took the partykit repo, and they haven't moved it. But now now like the main code base for party server, party client, and party sync, those all now live on on Cloudflare's own GitHub, and they're they're different packages.

• 28:26Scott Tolinski?

  Yeah.

• 28:26Wes Bos?

  Now, which I like, because like partykit was built on top of workers. You don't abstract anything away now when you're using party server. You can just add it to an existing Cloudflare worker, which is, I like that as well. You're not they're not hiding that you're using a worker like partykit was. So that's the server. It's obviously syncing state between all of the open browsers. But then there's also the aspect of like, how does it get to the the LED hardware? Right? So what's happening is

• 28:58Wes Bos?

  Part of the server, we are checking as frequently as we can for what are called dirty pixels, right? So data is coming in all the time. And then every now and then we run a function that then checks, what do we need to send to the microcontroller that's running WLED to tell it which pixels on this grid to now change? And the way that that works is there's a little bit of a diffing algorithm. It detects which pixels are dirty, meaning they have changed from what is currently on the LED grid. And then it creates just an array of all of the currently changed pixels, right? So if I were to just take red and paint like two or three of these,

• 29:44Wes Bos?

  it's going to say, okay, pixel 400, pixel 800, and pixel 650 have changed, and those are red. And then what it's going to do is it bundles that then up into a request, and then it will send that to the web server that's running on the ESP32 via the WLED software.

• 30:10Speaker 02

  So I guess to clarify, is that just like a separate process running on your computer? It's basically another party server client that has a direct connection to the grid? Because it's on your local network?

• 30:22Wes Bos?

  I actually did it both ways. And I'll tell you about the problems that it had. So what I'm doing and what you see right now is it's a function called checkforupdate. I guess I can just show you because we have the code here, right?

• 30:38Scott Tolinski?

  Yeah, show me.

• 30:40Wes Bos?

  There we go. It's a function. It's just a method inside my durable object called updateLED. And that will then run a bunch of code that checks if there are dirty indices.

• 30:56Wes Bos?

  And then if so, it will send it to the WLED. And I wrote a whole nother WLED library for... There's a whole bunch of logic as to how it gets sent over. I'll talk about that in just a second as well. And then that will just... That's a recursive function, which will try to... It will call itself as soon as it gets a response back from the WLED saying it has successfully updated the pixels. So that was the first way that I did it. And I'll tell you about the problems that I hit and then the second way, which you suggested. The WLED has several different ways to send data to it. There is a WebSocket API and there is an HTTP,

• 31:41Wes Bos?

  like just a normal API endpoint that you can hit with like a fetch request. I just call that the fetch APIs. It's simpler. Both of them have limitations. The WebSocket API is faster because you... When you connect to it, you have a single socket connection open. And then anytime you want, you just send updated pixels. And that is so much faster than the fetch-based API. However, you can only send, I think it's something like like 100 pixel change at once. So if somebody was to take like a Firefox logo and drag it on here, it would very quickly overwhelm the WebSocket.

• 32:25Wes Bos?

  And not so much that it would reject it, so much that it would crash this thing here because there's only so much memory in these microcontrollers. Not that it can't hold an array of 2,300 values, but the WebSocket and the HTTP API are JSON-based, meaning that the microcontroller has to take the JSON payload and parse it from JSON to whatever the C-based array is. And then it has to then send that data over GPIO via ones and zeros. So... The WebSocket API can't send too big of payloads.

• 33:07Wes Bos?

  And then the HTTP API can send something like, it says it can do 2400, but I tested it. I kept crashing it. I think I'm sending something like 800 or it's bytes. It's not pixels, but basically when I'm writing a full image, I have to send four or five consecutive fetch requests. Otherwise it will crash and reboot the entire microcontroller.

• 33:35Scott Tolinski?

  What happens when that crashes?

• 33:37Wes Bos?

  It literally just, the whole screen goes blank and then it reboots and then shows you the initial thing and then the WebSocket reconnects. It's not a big deal, like it recovers, but it obviously goes blank entirely. So I wrote a whole bunch of logic to say if this is a large, like I have a bunch of variables as to like limits. There we go. WebSocket byte limit, so 1400 bytes. And then HTTP byte limit is 5000 bytes. So that's, I can send, I don't know, four times, just under four times more via fetch request than a WebSocket. So when you're drawing like a little bit, this is all WebSocket.

• 34:21Wes Bos?

  But then when you, when you draw a lot of changes, then it will switch over to fetch and use those. So I wrote a bunch of logic in here to switch between the two. So that was my first two, and I wasn't really happy with either of those because I was, I was doing like the, the video and it was like, like one frame a second. It's really slow to actually draw. And I was like, man, like, that's kind of, that's kind of slow. And these things, these LEDs are not slow, right? You can, you can put like gradients on them and whatnot. They're pretty powerful. But what is slow is, I'm pretty sure is this, the JSON parsing and the WebSocket connections and, and having to run a web server. So ideally you would use a lower level thing

• 35:05Wes Bos?

  that would simply just send raw ones and zeros from whatever to it. And that's via something called a U. UDP and UDP is is not like like TCP UDP is a one way signal where you can just send commands and and you never get a response back from it. You just send, send, send, send. And that's how I did a video many years ago flying a drone with React. That was a UDP socket, just sending data from the React app

• 35:33Scott Tolinski?

  Right.

• 35:34Wes Bos?

  over Wi-Fi. That's a UDP sockets. But that still only allowed me to to do 400 pixels at once. So that was was still a bit of overhead and it had the the downside of Cloudflare workers. They have their runtime is not Node.js. Their their runtime is something called WorkerD. And they support most of Node.js. Part of what they don't support is the socket API. So you cannot open a UDP socket from a Cloudflare worker. It has to either be like a Web socket or an HTTP request. So I ended up making a bridge

• 36:16Wes Bos?

  that like like CJ suggested was I made a server side script that basically just joined this room and watched for changes. And as those changes happened, it would send them over the UDP socket. And then I ran into a whole bunch of trouble over like I was sending too many, you know, like if you draw like that was like 18 and then times I had 100 people in there.

• 36:44Scott Tolinski?

  Okay.

• 36:44Wes Bos?

  It was it was too much. So then what I had to end up to do is I would have to collect them as they're coming in and then shoot them out the door as fast as I could. At that point, I scrapped it and I said like people who do this LED stuff professionally, they send the raw data over probably not over Wi-Fi, probably not over some thing. They're they're sending it like raw over a GPIO pins, right? You're actually physically connected to it and you're sending the data. There's no parsing or anything. You're simply just sending it straight from whatever you want to to the value. And there's there is several other implementations aside from UDP

• 37:30Wes Bos?

  that WLED will implement. I just haven't had the had the time to to dip into it.

• 37:37Scott Tolinski?

  So is this as fast as you could get it refreshing with your current strategy?

• 37:42Wes Bos?

  Yes, yeah, this is, and in this case, right now, it is a little bit slow because I have OBS open.

• 37:49Scott Tolinski?

  Oh, yeah.

• 37:50Wes Bos?

  It's totally unrelated. If I were to close OBS, this would be butter. This is butter smooth when I'm drawing on it. And it's, it's pretty much real time, just drawing on it and then uploading a photo, drag and drop. It does it in two or three segments.

• 38:07Scott Tolinski?

  Who would have thought that OBS would have been the thing that makes me want to get a new computer? That's the only time I'm like, man, my computer sucks is when I have OBS open.

• 38:16Wes Bos?

  I can't even figure it out because my, my resources are not pegged at all. It's just something about opening OBS. It makes my, it's not even like, like, look, it scrolls fine. I can hover over stuff fine.

• 38:29Scott Tolinski?

  You open it and your computer just goes, dear God, no, no, close it.

• 38:33Wes Bos?

  My computer, my computer right now is butter, but there's something about browsers.

• 38:37Scott Tolinski?

  Graphics or what?

• 38:38Wes Bos?

  And the drag or the mouse drag drop, it's, it's jacking the mouse event somehow.

• 38:44Scott Tolinski?

  Something.

• 38:45Wes Bos?

  And, and they are very laggy. So yeah, this, if you're not impressed by this, it's, it's butter when I don't have OBS open.

• 38:52Speaker 02

  And I guess to clarify, that's because your server is running on the same computer as OBS. Like if this were running in the cloud, this would, you wouldn't have that limitation. Is that accurate to say?

• 39:01Wes Bos?

  No, no, the problem is that my literally, my mouse event in the browser is not being fired quickly enough to the thing. So it's, it's not a resource problem at all. It's, it's that my browser, and I've tried every browser on my computer, does not fire these events. What if I use my built in mouse? No, same thing. But it's, it's butter otherwise, but obviously it's not butter for, it's not butter for like uploading entire images.

• 39:33Scott Tolinski?

  Yeah, it's butter though, not butter though. Yeah.

• 39:35Wes Bos?

  Yeah, but that, that whole, like, I'm working on a project right now where I'm decking out our whole house in these things.

• 39:42Scott Tolinski?

  I love this. Wes is turning his whole house into an LED panel.

• 39:46Wes Bos?

  Like my house, it's gonna be like a gamer street, but I'm gonna deck out my whole house in these, which is. These are RGBW pixels, so they have a they have a white pixel as well, which allows you to get nice, warm and much brighter. Because if you've ever tried to use like an RGB light, you tried to do like colors, you'll notice that it gets much dimmer when you do like a purple versus just a flat white. But anyways, I'm putting these on my house, which I'm really excited about. And like the people that do this professionally, they have like hold like they have Elsa dancing on a grid like eight times the size of this, and it's like 60 frames a second.

• 40:28Wes Bos?

  So it's definitely possible, just not, you shouldn't be using a JSON API.

• 40:34Scott Tolinski?

  Yeah.

• 40:35Wes Bos?

  For that. Oh, also, another thing I didn't dip into, which is I'm only sending the dirty pixels. But one thing you can do, like if I were to draw from pixel three to pixel eight all blue, there is the ability to tell it, start at pixel three, go to pixel eight, make them all blue. So that would compress it a little bit. But even just looking at this like Firefox logo, there's, I guess there's like several instances.

• 41:04Scott Tolinski?

  But even then, what you're doing is you're, that's kind of a micro optimization when like the biggest optimization would simply come from

• 41:12Wes Bos?

  Just not using

• 41:13Scott Tolinski?

  A different approach.

• 41:14Wes Bos?

  Right. Exactly. Exactly. You're not going to get that big of a gain from that where we're switching over protocols would be much better. Opening up this, so this is running at local.wesbos.com. It's that this process is just running on my local machine, just so I can, I can get it going. And also it needs to run on my local machine because the ESP32 is also on my home network. And in order to send the data from my machine to that, they have to either be exposed to the internet, which I can do, or you just have it have it locally.

• 41:51Scott Tolinski?

  I have a question. Do you have any like not safe for work detection on here?

• 41:56Wes Bos?

  No, I don't. The first time I did it, I did it about six months ago, and I tried to get people to draw on it, and somebody did upload like a scantily clad photo, but

• 42:09Wes Bos?

  Nothing awful, but when I did the receipt detection, yeah, people are pretty, pretty good. The thing that people did when I opened it up is they started writing scripts in the console that would programmatically send pixel changes. And it was kind of fun because people were drawing like, you could tell that they were writing like nested loops to change the colors. And I thought, that's, that's pretty fun. But I think you could paint the, I don't know. Is there enough detail here that you could then send that to a, like a NS NotSaferWork detection library?

• 42:43Scott Tolinski?

  I think you could do a client side. There's got to be a client side detection for that.

• 42:47Wes Bos?

  Yeah, let me, let me show you what it looks like when I just upload a, an actual photo. Here, let me just find a photo of myself here. Like it's, like you can tell that's me.

• 43:01Scott Tolinski?

  Yeah.

• 43:02Wes Bos?

  That's, that's pretty good. Yeah, let me see if I can get the webcam stuff going real quick. Anyways, what I was saying about the, the real time stuff is that this is a VIT server running locally on my computer. And then I exposed that VIT server to the internet with a CloudFlare tunnel. And then basically people are visiting my, my computer from the internet without having to expose your, your local IP address, which was, was awesome. At one point, somebody wrote like a, like a thing to send like a million requests through. And my internet was getting kind of weird where it was just kept dropping in and out. But I just, I killed it at that point. I think they, they went away. You probably would have to put some sort of rate limiting in there. Here's the webcam code.

• 43:48Wes Bos?

  And the funny part about that is like, people were abusing the drag and drop images. So what I did is I just went to my code and commented out that feature. And I live updated on everybody who had it currently opens website. You know, like I can work on something locally and the, the VIT dev server was serving up tons of connections. It was proxying the web sockets, no problem.

• 44:10Scott Tolinski?

  It just asked me if I wanted to start my camera.

• 44:12Wes Bos?

  Oh, you try it, you try it.

• 44:19Speaker 02

  Whoa. Okay, yeah, so I have the app open on my computer too. This is, this is very realtime. Just on the, on the page itself, seeing his pixels.

• 44:30Wes Bos?

  And that's like, like literally it's sending the payload from Scott's computer to the durable object and then the durable object is then sending the update to both me, CJ and Scott before it actually paints to the thing. So it's, it's not bad. And I think if you were to, if you were to like, I don't think that the web socket is the, the hang up here. I like, I, sorry, there's two web sockets at play. There's one to the durable object and then there's another one to, to this thing here. Yeah. I think if I were to take the microcontroller out of the picture, I could get it to work pretty quickly. But I essentially am just recreating video and, and, and monitors. If you think about it.

• 45:16Speaker 02

  In your, in your demo video, that's one of the last things you said. And it's so funny because like, literally just buy an external HDMI monitor and put images on it. Like, you don't have to do this, but it's awesome that you're able to get low level to the pixel. And, uh, yeah.

• 45:32Wes Bos?

  Yeah.

• 45:32Speaker 02

  Love to see it.

• 45:33Wes Bos?

  That was pretty fun. The future, I don't know about the future. This was, this is a really fun project. I think what I would like to do is, is to hang it on my wall. And just to find a way for it to just be always open and put it on my website. And just have people come by at any time, draw on it. Or I, the other thing is that I can, I have admin controls where I can set which room is on there. So I can have one room, which is like forward slash syntax. And if I were to draw.

• 46:06Scott Tolinski?

  Yeah.

• 46:07Wes Bos?

  A syntax logo to this one, you'll see it's, it's not mirrored to the LED. But if I go to the admin, I could just switch it.

• 46:16Scott Tolinski?

  Yeah.

• 46:17Wes Bos?

  Now that room is being painted. So I could have like a syntax room. I could have a drawing room. I could have like a stats room. And then I could just have them cycle through those rooms. Or I could have buttons on my, on my little stream deck here that would switch them. There's just so much, so much fun stuff that you could do with it. But that was my project. That was, it was a ton of fun. I can't believe I get to talk about it for an hour. Uh, but I just love digging into all of this fun stuff.

• 46:46Scott Tolinski?

  Sick. Wow, that was awesome, Wes. I love it every single time. Nice work.

• 46:52Wes Bos?

  Cool. All right, let me know if you enjoyed that or if you have any suggestions on things I said add. Leave them down in the comments below and I guess we'll catch you in the next one.

• 47:01Scott Tolinski?

  Peace.