876: MIDI & Music in the Browser

Wes and Scott talk about the Web Audio and Web MIDI APIs, diving into how they enable powerful music and sound manipulation in the browser. They explore MIDI messaging, synthesizing audio, creative coding with music, and share hands-on projects, tips, and libraries to get started. Show Notes 00:00 Welcome to Syntax! 01:17 Brought to you by Sentry.io 02:04 Fun audio experiments Bebot 05:32 What is MIDI? Web MIDI API 14:18 Advanced examples with WEBMIDI.js WEBMIDI.js 17:02 Outputting MIDI messages 24:40 Exploring the Web Audio API webmidirtc 31:20 Audio sampling in the browser 37:35 Media Recorder 39:21 Fun projects MIDI chord machine 42:08 Sick Picks + Shameless Plugs Sick Picks Scott: MX Master 3S Wes: Microcontoller Shameless Plugs The MOST Starred JS Projects Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott: X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Scott Tolinski?

  Welcome to Syntax. On this episode, we're going to be talking about the Web Audio API and the Web MIDI API, what these things are, why they matter, and maybe some interesting things about them that you might not know. Because me personally, when I think of Web Audio and Web MIDI, I'm thinking of maybe playing sounds or accepting input from MIDI. But you can do a lot with these things. And there's a handful of really great community libraries out there to help you make both music, but also just control things with MIDI messages. So we're going to be talking in this episode about some really interesting audio technical concepts, such as sampling. We're going to be talking about MIDI in general,

• 0:44Scott Tolinski?

  the types of messages that are sent, how it's sent, what it means, how we can interact with that in the browser. And then ultimately, I'll be showing you some really neat demos of stuff we can do. In fact, my desk-

• 0:54Wes Bos?

  Play us in, Scott, will ya?

• 0:56Scott Tolinski?

  Play a synth, okay.

• 0:58Wes Bos?

  No, play us in.

• 1:00Scott Tolinski?

  Play us in.

• 1:01Wes Bos?

  Yeah. Sounds awesome.

• 1:04Scott Tolinski?

  Okay. I can't hear that, so I don't know what it sounds like in my monitor right now. But I'll be definitely showing you some of this in just a little bit. I'm building something fun with this, so I thought it'd be a good opportunity to take a time and do an explainer. But hey, if you're trying out new things in your site, maybe you're pushing it live, and maybe that new thing is broken. Maybe it takes down your site. Maybe you're getting a ton of issues on one specific thing, or perhaps I think that I have covered all the use cases that I thought I should cover in this, and it's not working. So you're going to want to use software like Sentry. At sentry.io/syntax, sign up and get two months for free. This software allows you to keep track of all of the errors, bugs,

• 1:45Scott Tolinski?

  issues, anything that's happening in your site, fix performance issues, find out what went wrong. You even get a session replay, so you can see a video of what happened. Let's say the user went to click on a keyboard on a website, and it played the wrong note. I could figure that out with Sentry. So check it out at sentry.io. Wes, my man, what's happening today, man? How you feeling?

• 2:07Wes Bos?

  Pretty good, pretty good. Excited to talk about Web Audio. Dabbled in it, I don't know, three or four times. Yeah, there's this app called BeBop, which is on the iPhone. I've had it for probably like 15, 10, 15 years. And it's generated sound and it goes up and down the scales. And I don't really know a whole lot about music and how all of that works. So I went and built that at one point.

• 2:30Scott Tolinski?

  Oh, did you? Cool.

• 2:31Wes Bos?

  HTML, CSS and JavaScript. And it was kind of a fun learning experience.

• 2:35Scott Tolinski?

  Yeah, so what that on BeBop, I believe it's the vertical. If you're holding it sideways, the vertical is the pitch or it's the vault, the, the tone or something. One of them is the pitch and one of them, the tone. It's kind of like a, a theremin and, and an access of how you're doing that. Yeah. I have a lot of experience with this stuff. I did a lot of music programming when I was in college. In fact, I took several years of a music programming class where I was doing like complex Markov chains to generate music, generate songs, like, you know, automatically generated unique songs. Every time we were doing things using physical hardware triggers to trigger music. I built a synthesizer out of a projector, actually, Wes. That was like one of my coolest projects.

• 3:19Scott Tolinski?

  What I did is I projected, this is actually a really technical, like interesting way that I solved this. I projected color beams of different color onto the wall. So let's say there's like six different colors, red, whatever, whatever. And I projected that and I had that image saved of what that should look like. And then I had a video camera pointed at that. And using a, or a Max MSP, I was able to overlay the two. So I had what the video was showing of the projector plus what it should look like. And I was able to do a, if one of these color beams suddenly has a difference of enough threshold, then fire a MIDI note. And that was sent to a controller to play a sound at a certain tone.

• 4:03Scott Tolinski?

  So you could run through this thing and it would sound like you're dragging your hand through a keyboard. It was probably one of the cooler projects that I put together.

• 4:09Wes Bos?

  So that would be a fun canvas project. If anyone's looking for something is take a video and then figure out, you probably just use the X, Y coordinates of the person's hand. Or you could, you could overlay it if something is breaking up a specific color.

• 4:25Scott Tolinski?

  Yeah, well, let me tell you, I even did a theremin, which you don't know, theremin is used by like radio waves, I believe, and you put your hand into it and it detects where your hand is and you go

• 4:35Wes Bos?

  Yeah, it's really cool.

• 4:37Scott Tolinski?

  I made one of those with my webcam and I did video tracking. So I would hold up a little color square and where the, where I put it and how big the color square got, I had like a 3D dimensions of being able to tweak both the, the audio, the quality, the timbre, all that sort of stuff. So I was able to make noises with my webcam just like that. So, long story short, I've been, I've been experimenting with just tech and music, the intersection for a long time. And, and actually, since getting into web stuff, a lot of that kind of fell off for me for no reason in particular. I don't, I just became less interested. But more recently, I've been getting back into some of this stuff and the web audio APIs that exists are really fantastic for a number of reasons.

• 5:21Scott Tolinski?

  I had used the web audio API to do audio synthesis. I'd done media recorder. I've done kind of like things here and there, but I haven't really dove into it until somewhat recently here. So, we can start off first and foremost, let's talk about the web MIDI API, because I think the web MIDI API is fairly underrated amongst folks. In fact, even people who know what MIDI is and understand MIDI, might not understand what you can do with MIDI in the browser. So, first and foremost, MIDI stands for Music Instrument Digital Interface. It's the way that electronic music instruments can interface with devices,

• 6:00Scott Tolinski?

  computers, those types of things. And when you have a MIDI device, it is sending messages. It's sending messages to whatever the device is and the device, the digital audio workstation, you know, you have Logic, Ableton, those types of things.

• 6:15Wes Bos?

  Yeah.

• 6:16Scott Tolinski?

  Receive those MIDI messages and those MIDI messages, unlike an audio form, like when you send a waveform, that is

• 6:24Wes Bos?

  Yeah.

• 6:24Scott Tolinski?

  audio. When you send a MIDI message, it's just Data, and it's not even data of how it sounds. It's a few messages and typically those messages will look like, play this note, play it at this velocity and start. Or, you know, note on this note, this velocity.

• 6:46Wes Bos?

  So it's just a description of the notes and how long and how high they should play them, right?

• 6:54Scott Tolinski?

  Correct. And there's all kinds of things that people can use MIDI for beyond notes. Like I have a MIDI keyboard here and you can do like pitch bend and modulation with these little touchpads. Those are all sent via MIDI or even like being able to change volume with these knobs. That message is sent via MIDI as well. But MIDI can send, you know, it's more than just like you think about like a key press on your keyboard. You get the on and off, but you don't get velocity, you know? You don't get how hard you pushed it, which that's a lot of interesting information, right?

• 7:29Wes Bos?

  Yeah, because BetterTouchTool, like the keyboard shortcut and everything tool that I use, they have full support for a MIDI controller. And I always thought like, oh, that's kind of interesting. People are hooking up keyboard shortcuts to their P L. And it makes a lot of sense because you can, you can store a lot more information in a MIDI payload, I guess it's called. You're going to go into its three bytes, right?

• 7:55Scott Tolinski?

  Yeah. So most MIDI messages in the typical sense you're playing a keyboard are three bytes. It can also be two or one depending, it can be more than that I've heard. I don't really, if you're like a MIDI expert out there, in general, when you're working especially with like a keyboard, it's going to be three bytes. The status byte is eight bits and then the data bits or data bytes are seven bits each. So the first message, the first number that comes in is an eight bit byte of the status. That status typically means, if you break it down into actual binary, the first digit is telling MIDI that it's a status byte. The next three are telling it if it's a note on or a note off.

• 8:39Scott Tolinski?

  Like you're either pressing it or you're telling it to stop, right? And then the next four are just simply what channel it's on. So that's the status because you can send MIDI messages on many different channels and receive them on multiple different channels as well. That allows you to-

• 8:55Wes Bos?

  What is a channel?

• 8:57Scott Tolinski?

  So a channel is exactly kind of like what you think about it. It's kind of like a pipe. So if I'm sending this message down pipe number one in my controller or wherever I'm receiving that message, I can look in pipe one and I can know that this is coming for this specific thing, i.e. like you have this thing, right? It has drum pads and it has a keyboard. I can put this on one channel and when I send a message here, it could send that in channel one. And then the MIDI receiver can listen and say, anything that's on channel one, that's a keyboard. And then I could send these on channel two. Anything that's on channel two is going somewhere else. Maybe it's a drum, maybe it's modulating something, maybe it's changing something.

• 9:36Wes Bos?

  Oh, okay. I see, I see. Yeah.

• 9:38Scott Tolinski?

  Yeah. Yeah, it's like just putting things in different, you go here, you go here, you go here type of way. So the second byte is a data byte and that's a seven bit. And by seven bits, it goes zero to 127. So this is the note. So when you're playing and I hit middle C, it sends a note on, which is 0x90. And then the second byte is going to be 60 for middle C because it's 0 through 127. And then the last one is the velocity, 0 to 127. And that's also a seven bit byte.

• 10:15Wes Bos?

  Can I tell you how hilarious this is that we both got into writing binary within the last seven days of our entire career? So I recently did a video on controlling my Roomba with JavaScript. I hooked up like a serial port to it. There's three different like motors on it. There's a spinner motor on the side, there's the actual vacuum motor that sucks, and then there's the main brushes underneath that spin. And then also two of those brushes can spin in reverse. And in order to send the kind of state of where you want the motors to be on, right? Because there's five different things, there's a million different combinations, not a

• 10:56Wes Bos?

  million different combinations, but whatever the permutation of that is. And in order to send that combination, you send it over in binary, right? And I thought it was so hilarious that like I have never actually written raw binary or bits and bytes like that before. And here we are doing it in the same week.

• 11:18Scott Tolinski?

  It is funny because I was, like I mentioned, I've been doing quite a bit with this. And at first I was trying to do everything with the bits and bytes. I was trying to like really get into the raw MIDI. I was like, yeah, I'm just gonna use MIDI straight up and I'm gonna be very like uh, content with that. But the more and more I got into actually uh, playing notes and playing scales and wanting to do like more real uh, audio stuff with it, I realized like, oh man, I, I really don't wanna be getting into that. I want, what I wanna be doing is just worrying more about the audio stuff. So actually, I'll, I'll share some libraries in a little bit that, that help with that sort of thing. Because there is like music-based MIDI libraries, or somebody's already figured it all out.

• 12:00Scott Tolinski?

  You can say, "Play this note," and it will just play it rather than having to worry about the binary. But yeah, I, I got into it quite a bit. So basically what happens, like you play a keyboard, it sends a note on, that note in the velocity. And then the moment you release the key, it sends another message, which is "Note off." And um, if you wanna see this in action, I'll actually uh, share my screen here if you're watching on video. And for those of you who aren't watching, we, we'll just basically tell you a little bit about what's coming in here. So basically what I'm doing in this example here is I'm getting access to my MIDI inputs, and then I'm setting a function to the "on MIDI" message. This message, uh, this function just receives a message. You can see that anytime I hit a key, and I know if you're just listening to this, we'll describe it,

• 12:45Scott Tolinski?

  but you're getting, it's a straight-up JavaScript event that gives you some information. Inside of that information uh, is a data array. It's all kinds of stuff. You have the array buffer here, uh, you have what type it is, it's a MIDI message. Um, you can come in here and see exactly like what exactly it's coming in from. So for instance, we, the device itself is, this is a Samson, it's just a crappy MIDI keyboard. Uh, so it tells you, "Hey, this is coming in from a Samson, Samson MIDI keyboard." And the input message itself comes in as message data. And that gives you the command, the note and velocity inside of the message data.

• 13:27Wes Bos?

  That's cool. How does it know what the keyboard is? I'm assuming the hardware sends along like an identifier byte? And then the library looks that up?

• 13:38Scott Tolinski?

  Oh, I'm sorry. This is not done with the library. This is done with the MIDI API itself. So my understanding-

• 13:44Wes Bos?

  Oh, this is not even through a library?

• 13:46Scott Tolinski?

  Yeah, this is just the device itself sending that information to the browser. Because you'll see here in my other example that you can check to find all MIDI input and output devices on your device just by giving the browser access to MIDI.

• 14:03Wes Bos?

  Oh, I see. I see. So the browser itself has a list of... Similar to how you can get all webcams and microphones. You can also get all MIDI devices. Okay.

• 14:13Scott Tolinski?

  Yeah, so I have all my MIDI devices, all that stuff. Either way, that's what type of message it's sending. Now, in my more advanced example, I did use this library called WebMIDIJS, and it's super good. I'm gonna tell you that right now. If you're doing anything with this stuff, the WebMIDIJS is well worth using instead of potentially trying to do it raw yourself. Because, man, dealing with the raw information is kind of tough. Especially if you're doing music stuff with this. So with this library, I'm able to do a more standard, like, add event listener. So I'm listening for a note on and note off and control change, these types of events. And this time, when I hit a key,

• 14:58Scott Tolinski?

  I get a lot of information with each one. Like, the attack is the velocity, the release, the octave.

• 15:06Wes Bos?

  Attack, that's cool.

• 15:07Scott Tolinski?

  Yeah, so it gives you more information. Like, it's giving you the name of the note itself. Because this library is taking... It's like, we're gonna decode the MIDI messages into things that make sense. So if you're actually doing music with it, you'll wanna use MIDIJS, or WebMIDIJS, I believe. And, man, what a cool library for that.

• 15:26Wes Bos?

  And the velocity, or the attack, as they call it here, that's how fast you hit the key?

• 15:34Wes Bos?

  So I imagine somebody playing piano and if you hit it very quickly, it would sound different than if it was you pressed it slowly, right?

• 15:42Scott Tolinski?

  So yeah, I just smacked it and it gave me a one. Now I'm going to lightly press it.

• 15:46Wes Bos?

  As fast as possible.

• 15:48Scott Tolinski?

  Yeah, I lightly pressed it and it's giving me a zero, a point 0078. So yeah, if you're playing lightly or you want to, you can do all kinds of stuff. And you know, it's a more advanced digital audio ways of working with MIDI West is that they're actually checking that velocity. And even playing a different audio sample based on that velocity, like you have a snare drum. And that velocity, if it's below, you know, point five, it's like a little a recording of a tap of a snare drum and then you hit it more, it actually plays a different audio file. Like a snap, you know, because oftentimes what we're doing with MIDI is we're using these to control and play waveforms or generate synth. You're generating synthesizers or something like that.

• 16:29Scott Tolinski?

  But typically, you know, if I hit this and I'm doing a drum kit, it's usually just a waveform of a drum being triggered in the background. So the MIDI message is like play this note, whatever it's like fire this MIDI or fire this waveform. Yeah.

• 16:43Wes Bos?

  That's cool.

• 16:44Scott Tolinski?

  Yeah. So MIDI really neat, right? You can do so much with it. What I did not realize about the MIDI API is that it doesn't just allow you to receive MIDI. Cause that's the way I always perceived it, right? You have MIDI devices, the web browser can accept those MIDI messages. You can do something with it, but the browser can also output MIDI messages, which I found to be really super neat. So for outputting MIDI messages, your browser, just like inputs, has a set of default or your, your computer has a set of MIDI outputs. And by default, it could be any number, one of these things. You might not have any, to be honest, if you don't have any of these things set up. So for instance, I have my audio converter, which is my digital audio interface that shows up here.

• 17:28Scott Tolinski?

  I have the keyboard that actually shows up as a MIDI output as well. And then I have this IAC driver bus. So you can do this in Windows or Mac. I don't know the Windows way to do this. But in in Mac, you go to Audio MIDI Setup. You go to Window and you open up MIDI Studio. MIDI Studio gives you access to like an interface. You click on IAC Driver. You enable that thing. And then what that does is it essentially sets up a MIDI bus on your computer that you can send audio to. So it creates a MIDI device on your computer. And by doing that, that now shows up in browser as a place I can send the MIDI. So I'm sending this MIDI to this bus.

• 18:13Scott Tolinski?

  And then in another application, for instance, a digital audio workstation, I use Ableton Live a lot. You can then see the one that you created and you can use it here. So for instance, I built this keyboard synth. So if you're watching along, you can see this. Otherwise, you'll just be able to hear it. I built a keyboard synth and allows me to do programmatic music. So what's cool about this synth here that I built is it's just sending MIDI messages, right? But these MIDI messages are actually able to do a lot more than just like a straight keyboard. So like a normal keyboard, you play the notes. But what if you could play chords?

• 18:58Scott Tolinski?

  So I have it set up. Anytime you play the A key, it plays an A chord. And that's sending three messages, MIDI messages. So if I'm playing and recording in Ableton and then, oh, shoot. This is the hard part because you have the browser active. So let's start the recording. I hit the A key. Ableton receives those messages and records them right here as data.

• 19:27Wes Bos?

  This is blowing my mind because. MIDI is a way to get data in and out of the browser. It's it's almost like a web socket, isn't it?

• 19:39Scott Tolinski?

  Yes.

• 19:40Wes Bos?

  Like, you you could use MIDI as a way to pipe data out of the browser. And then you can use, like, you could have, you could probably have like a note app that listens on for on like a local MIDI, that that MIDI source that you just created. So you're sending it from the browser over MIDI and then you could pipe it into could obviously could be a music app like you're doing here, but it could be like BetterTouch Tool, which is like custom keyboard events. Or it could be like a note app that you're sending data over. It's probably the the data is limited, right? You have three bytes. But you could.

• 20:18Scott Tolinski?

  Yeah, and what's cool about it is you can augment it. So for instance, when I'm hitting this key down, I'm saying, oh, play the A key, but also play the the D key, also play the G, right?

• 20:32Wes Bos?

  Yeah.

• 20:33Scott Tolinski?

  But you can do some math on that. So I built this, what I'm calling a locked chord view. So now, if I play the second row of my keyboard, it's always going to give me like a chord within that scale. So if I select a C major scale, every chord I play automatically is going to remain in the C major scale because I'm I'm doing that math wise. And so I added some some keyboard controls of this where I can I can change which key it is.

• 21:16Scott Tolinski?

  And you can change major or minor. Also, in the normal chord view, you can modulate them. So I have a second row where I hit a chord and then I can make it minor

• 21:30Scott Tolinski?

  just by hitting a two key. And then I hit the three key, it makes it a seventh, a major seventh, a minor seven, diminished.

• 21:41Scott Tolinski?

  Yeah, so you could modulate it, you could do all kinds of stuff with math just by changing the MIDI notes on the input. And then furthermore, you can use things like Ableton Live to augment it even further, where I'm piping in an arpeggiator. And let's change this to locked chord mode.

• 22:04Scott Tolinski?

  And now it's playing a chord into Ableton. Ableton is spreading the notes out.

• 22:11Wes Bos?

  Beautiful.

• 22:14Wes Bos?

  I'm so inspired right now, this music.

• 22:16Scott Tolinski?

  You're just messing around with it, yeah.

• 22:18Wes Bos?

  This is, this is actually fascinating to me because it's like one of these overlapping things, you know, like there's web development, but then there's also music, but then there's also math, which is figuring all this stuff out. It's, that's really cool.

• 22:36Scott Tolinski?

  Yeah, it's neat. You know, I think the coolest thing about it for me is that now I'm able to build like a little exploration engine where I can like, I want to say I'm going to build a song in D minor, and then I don't have to like memorize my hand positions because I took piano, I was a music major, I know how to play all the scales. But like, sometimes I'm just like, oh, what was that scale again? Oh, you know, I'm in a weird key or whatever. I don't, you know, play it often, my hands aren't used to that. Now I can just be like, hey, give me the, the fourth chord in a D, D major scale and not even think about what the chord is, and it just gives it to me. So, pretty cool. So yeah, a lot of neat stuff going on here. Overall, I think there's a lot of just opportunity here, and I'm going to take this a little

• 23:20Scott Tolinski?

  bit further and try to build something like truly that I can mess around with. I even thought about building the arpeggiator directly in the browser, and then I was thinking about, I was like, man, Ableton does that way better than me trying to write my own arpeggiator, because then I got to write a metronome and a clock to keep it all like in sync and stuff. I don't want to get into any of that. So, that's WebMIDI.

• 23:41Wes Bos?

  Man, now you have me going down, downloading a ringtone.

• 23:45Scott Tolinski?

  Mm hmm.

• 23:46Wes Bos?

  Because ringtones are MIDI, right?

• 23:49Scott Tolinski?

  They were.

• 23:51Wes Bos?

  Well, yeah, they were, yeah.

• 23:52Scott Tolinski?

  Yeah.

• 23:53Wes Bos?

  20 years ago, but I always wondered about that. I'm like, how come you could just download ringtones back in the day? And it was because it was just a file with a description. So it was like a 1KB file with a description of how to play them. And then the phone itself is the one that actually has to turn it into to music, right? Which is why it sounded so funny.

• 24:18Scott Tolinski?

  Yeah, it uses the same audio engine that they use when you pushed any of the keys. Like beep, boop, bop. It's using that same audio engine to recreate the songs. That's MIDI, man. It's so cool because it's small. MIDI is really small, where audio in waveforms are really big. So it's very transportable. It's fast to send. It's fast to do anything with. So one of the many cool benefits of MIDI. Let's talk about the Web Audio API, though, because this is the Web MIDI API. The Web Audio API can be used for a ton of interesting stuff, too. For instance, you can generate sounds using an oscillator. An oscillator is kind of what it sounds like. It creates a waveform, right? So you're creating like a sine wave and then you're controlling the hertz. What is that, the frequency?

• 25:00Scott Tolinski?

  You're controlling the frequency of the hertz and that changes the pitch. So the velocity is how big the waveform is. And the pitch is how tight or wide the waveform is. And if it's a sine wave, it's going to sound one way. If it's a square wave, it's going to sound more buzzy, sawtooth, those types of things. There's all different types of waveforms you can do for that. But you can generate audio straight up in the browser using an oscillator. And I'll show you an example of that in just a second. You can also bring in sound via WAV files. You can bring in sound via the WebRTC media stream. A lot of neat stuff here.

• 25:38Wes Bos?

  Yeah, so one little demo I built a while ago when I was looking into the the web audio API was creating a dialer with your phone. So you know, when you when you hit the numbers on a phone, it goes

• 25:53Wes Bos?

  So it turns out that all of those sounds are and I initially stumbled upon this when I was watching a documentary on phone freaking.

• 26:02Scott Tolinski?

  Yeah.

• 26:02Wes Bos?

  Which is how how who was the guy that did?

• 26:05Scott Tolinski?

  Kevin Mitnick?

• 26:06Wes Bos?

  Kevin Mitnick, how he

• 26:07Scott Tolinski?

  But also Steve Wozniak did a bunch of phone freaking.

• 26:10Wes Bos?

  Yeah. They basically they blew a whistle they got in Captain Crunch and that whistle made the same frequency that would allow you to bypass these like phone companies and whatnot. I thought it was kind of interesting. So I like I looked into it and I was like, you can make any audio you want in in the browser with the the web audio API. And I I went into the Firefox OS codebase. So for those who are a little bit younger, probably about 10 years ago, Firefox launched a operating system that was meant to run on your phone.

• 26:48Scott Tolinski?

  Yeah.

• 26:48Wes Bos?

  And it was written entirely in JavaScript. And it was I was at the launch of it and they gave us they gave us like an Android phone that had been flashed with Firefox OS. And it was really interesting. And what was cool was that you can you still can you can go into the codebase for Firefox OS. And you can see how they implemented all the like native apps. And one of the native apps was the dialer, literally dialing stuff. So I went into it and they have an array of the frequencies that are needed to play because when you press a button on a dialer, it actually plays two frequencies at once. And every single number has a different frequency. And that's how that data can be sent over the phone.

• 27:33Wes Bos?

  And I just like I re-implemented it in, I don't know, maybe 20, 30 lines of code of JavaScript. It's just creating two oscillators and then setting the the frequency of them when you press down. And then when you let go, you turn them off.

• 27:48Scott Tolinski?

  Yeah. So I don't know if you'll be able to hear this because it's in the browser. Let me turn on my PC, maybe that will help. So basically, for those of you watching, I have some code on the screen. For those of you listening, I'll describe it. You create a new audio context with window dot audio context. Then you create a new oscillator with audio context dot create oscillator. So that creates essentially your oscillator. You set what type it is, is it a sine wave? So you say sine. Then you can set the frequency and you set a value at a current time. And this is a straight up Hertz number. So like 440 is A4. It's the first A key, or it's like the middle A key. So you go to middle C and you move over to A. That's A440. And then you connect it to a destination like your speakers.

• 28:32Scott Tolinski?

  You hit start and then you can stop it. So I have this code that you might be looking at. I'm just gonna hit enter on this and I don't know if you could hear that. It was super loud in my headphones.

• 28:42Wes Bos?

  No.

• 28:43Scott Tolinski?

  Blew my ear out. Either way, it played a sound tone. We'll have this code in the show notes. You can copy and paste it, throw it into your console, just hit enter and you'll get a sine wave. And you can see really quickly what in like six, seven lines of code, you're able to generate a sound using a sine wave directly in your browser with like no effort. And this can be used to do a lot of stuff, to create full synthesizers. That was one of my first projects was to create a keyboard that played audio synths just based on whatever key I'm hitting on the keyboard, similar to what the project I'm showing you in this podcast, but like for generating synths instead of sending MIDI somewhere else. And that's really great. I think it's really neat. There's actually a really great library that can take care of this as well called ToneJS.

• 29:27Scott Tolinski?

  Tone JS is super good. It handles like, just like Web MIDI JS handles a lot of the nitty gritty stuff. This really just makes it super easy. Play this note, C4, you know, you know, play it now, start out. Like it's way a nicer API than having to get into some of the nitty gritty of this all. But definitely a cool thing to mess around with if you're interested in actually generating audio synthesis.

• 29:51Wes Bos?

  Like even another use case, if you're not into music, sometimes in your app, you need to just play a little sound. Like for example, if something finishes and you need to play like a little boop or something is incorrect, like if somebody tabs away and you need to play an audio alert that something happened. Sometimes it's easier to simply just write.

• 30:14Wes Bos?

  What, seven lines of JavaScript to create that little boop and play it versus having to load a whole wave element and then like load it up, play, because the thing about the audio API and playing audio in the browser is you have to latch on to a click event before you can actually play audio. So you gotta like wait for somebody to like click something and the first time they click, you gotta set it up. And then you, you can play it later on.

• 30:44Scott Tolinski?

  Why is that Wes?

• 30:45Wes Bos?

  Nice.

• 30:45Scott Tolinski?

  Do you want to explain to the audience what that is?

• 30:47Wes Bos?

  Well, you can't just autoplay audio on websites because it's, it's the same with video, right?

• 30:53Scott Tolinski?

  Yeah.

• 30:54Wes Bos?

  Autolaying, those can be extremely annoying. So you have to do it. There's a whole, maybe we should do a little show on that, like on what makes you allowed to autoplay, because some, some websites can. And there's like an algorithm that goes into whether you're allowed to autoplay audio or not.

• 31:09Scott Tolinski?

  It's honestly kind of a mysterious algorithm. So yeah, no, super interesting.

• 31:13Wes Bos?

  It's just math, but yeah.

• 31:15Scott Tolinski?

  Well, no, it's that's, you know what, Wes, sometimes math is mysterious.

• 31:19Wes Bos?

  Yes, that's true.

• 31:20Scott Tolinski?

  Yeah. What's interesting is that the browser also takes care of like sampling audio. So those of you who don't know too much about how digital audio works. So let's say you have a record, like a vinyl record and you're playing that record. That is actually playing a straight, continuous waveform, right? That's playing just straight up, why, why it's called analog audio, right? It's playing just a straight up waveform. Where the difference between analog and digital, and it's some of the reason why people say they like their records better than their MP3s, is because when you have an MP3 or you have a digital audio file, kind of inherently what it's doing is it's sampling the audio.

• 32:02Scott Tolinski?

  So every X amount of milliseconds, it's taking just a single chunk of that waveform. So you could imagine like a smooth waveform suddenly becomes like stepped blocks, right? And the amount of stepped blocks, how finely you chop that, chop that up, we're, we're, you know, we're chopping carrots here. How finely you chop that up is what's called the sampling. So the higher the sampling rate, the more close it is to analog audio and the better it's going to sound. But also, the larger this audio form is going to be, the more data it is, right? Because you're sending way more data every single time you have smaller chunks.

• 32:42Scott Tolinski?

  So that's a big reason why people say digital audio doesn't feel as good as analog audio. Because analog audio is not losing any information where digital, you have to chop it up. So like, if you zoom in close enough, it's always going to be stepped. It's always going to be chopped instead of a smooth, continuous waveform. So if you ever hear sampling rate, that's what that's doing. You can control sampling rate in the browser in an audio buffer. So if you're sending audio over, you know, what, like over WebRTC, you're sending audio to somebody? And you want it to send less data, what do you do? You lower the sampling rate. The audio is going to sound tinier, it's going to sound more hollow, it's losing some of that information.

• 33:27Scott Tolinski?

  But it's going to be much smaller, right? So that's one of the reasons why, you know, when you record or we do this stuff, we want to record locally to get the best sampling rate instead of whatever we have to send over the network for West to hear.

• 33:42Wes Bos?

  Ah, interesting.

• 33:44Scott Tolinski?

  Yeah. So yeah, you can, you can control sampling rate directly within the audio buffer just by changing the sampling rate. Pretty neat.

• 33:51Wes Bos?

  I just found another project from Phil Miller. He's the dev behind Domino, D-M-N-O, doing like some like secret stuff. I just randomly found a project by him, which was WebRTC and using WebMIDI.js. So being able to play your MIDI things, but with friends. And instead of sending audio to friends, you're sending MIDI data. And then you're just reproducing it on the person's machine. So like, that would be, that's not a lot of overhead at all.

• 34:26Scott Tolinski?

  I was thinking about doing that same thing, so that's interesting to know that somebody else is on that.

• 34:29Wes Bos?

  That's so cool.

• 34:30Scott Tolinski?

  Yeah, I know, I was thinking about like how you could have a multiplayer MIDI workstation, or even you're recording MIDI in the browser multiplayer, or something like that could be really super interesting. I'm gonna check this project out, that looks really super neat. I should have actually looked before we recorded this, because I was just like, oh, I wanna make that.

• 34:48Wes Bos?

  No, sometimes you have an idea, and it's like, I don't even wanna look if somebody's done this, because I'm so excited about learning and doing it is, I don't wanna be like, ah, someone already did that, or I don't wanna see how they did it, because I love the process of trying to figure out how to do it myself.

• 35:06Scott Tolinski?

  Yeah, that's actually like a lot of what inspired this keyboard that I made for myself, because I got an ad, I got served an ad for a synth created by, or a MIDI controller, I don't even know if it has synthesis on it, I think it's a MIDI controller, from Tame Impala, there's a band, Tame Impala, they created their own idea generator synth, I don't even know what they're calling it. Regardless, it's sold out, they had like a limited number of them, and it does exactly what mine does, where you can hold one button to do keys, and hold another to modulate, or play arpeggios inside of a chord, or strum them, or do all kinds of stuff. And I went to look at it, I was like, oh, that sounds neat, I would love to own that. And then it's like, oh, we had a limited amount and it sold out. I'm like, you know what, I'm gonna build in JavaScript, so.

• 35:52Scott Tolinski?

  I had a buddy of mine message me on Instagram, and he was like, hey, did you get inspired by the Tame Impala synth to do this? And I was like, oh, shh, you know, yes, I did, that's definitely what I'm doing, I'm recreating it.

• 36:04Wes Bos?

  One more, one more neat fact, sorry, I keep just interjecting on the show with my neat facts.

• 36:09Scott Tolinski?

  No neat facts, I love it.

• 36:09Wes Bos?

  But this is my app where I built the dialer. And it wasn't working earlier, and I was just debugging it a second, and it says like, an audio context is not allowed to start.

• 36:23Scott Tolinski?

  Yeah.

• 36:23Wes Bos?

  Because it has to be attached to a gesture, right? We just talked about, you have to click first. But guess what is considered a gesture on Safari on iOS?

• 36:34Scott Tolinski?

  Uh, could it be like the accelerometer?

• 36:38Wes Bos?

  No.

• 36:39Scott Tolinski?

  No, I don't know.

• 36:40Wes Bos?

  The mute button on the side. I, I couldn't figure out, I'm like, it was working, now it's not. And so I turned, I had my mute button off. And I turned it off and then I turned it back on again. And then that was enough to consider it a gesture. And, and now Safari lets you play the audio, which is awful.

• 37:03Scott Tolinski?

  You know what doesn't need a gesture is the MIDI API. So that means like you hit a key and it would still register the MIDI event. And that, what that considers it a gesture? So maybe you could start audio synthesis then? I don't know, I haven't even tried that.

• 37:14Wes Bos?

  Yeah, you have, you have to wait for the first user input. Whether that's like hitting enter or, I'm curious what the gestures are considered now.

• 37:24Scott Tolinski?

  Yeah.

• 37:25Wes Bos?

  That's a browser level implementation. But clearly flicking the mute button on the side of an iPhone 13 Pro Max is enough of a gesture to allow it.

• 37:35Scott Tolinski?

  Yeah, I guess that gets us into the last little bit here, which would be the media recorder. Now, we're not going to spend a ton of time, but media recorder, as far as web audio APIs go, is really great for a lot of reasons. Like potentially you could be just building a simple recorder where it's connecting to your audio source, your microphone, you're talking into it, you can record that. You basically, you create a new recorder and you pass in data into chunks. And then, once you're done, once you stop the recording, you can then create audio directly in the browser. And it's pretty amazing. Like I can essentially record audio directly in the browser, save it as a file and then download that file directly entirely in the browser

• 38:20Scott Tolinski?

  without any additional software, just using the media recorder API. So, man, so much cool stuff in this, these APIs that I feel like are fairly underutilized, whether that is the media one, the audio API. So much cool stuff here. And that's not even getting into like WebRTC or any of that cool stuff that you can do there. So a lot of interesting things you can do. If you're a music person or just curious about any of this stuff, spend some time just diving into the MDN docs. We'll have a link in the show notes or fire up Tone.js or WebMIDI.js and experiment with this stuff because this is the type of thing that when I work on it gets me super inspired to make fun things rather than, oh, I'm just pushing

• 39:05Scott Tolinski?

  pixels on a website. No, this is like, I'm creating music. I'm making something that has actual sound. I'm, you know, something that I can show my kids, hey, push this key. How does this sound? And now push this minor note. How does that sound? Like, I could use these as educational tools. So very interesting.

• 39:21Wes Bos?

  If you're feeling at the very least burnt out on coding, putting buttons on screens and clicking them and updating databases, try hook up a friggin keyboard or a Roomba to your browser. That's really fun. And it will ignite. Not to say that I'm burnt out. A lot of people thought that on my video, but I always recommend people to do creative coding and go outside of your, your regular button clicking because it makes you so excited about web development in general.

• 39:55Scott Tolinski?

  Yeah. Imagine this, right? Here's some, here's some freebie project ideas. Like, we, we talked about sending and receiving messages or data or audio or stuff like that or generating. You can use that information in all sorts of cool ways. Like, what if you took the MIDI message? I hit a specific note and that note controls the background color on the website. That's a fun little project right there. I hit this key, it changes to this color. And you could just, what, clamp the, the notes on a scale to the range of RGB colors and, you know, have some fun really just doing something expressive or creative. Or I got a MIDI drum set. I'm gonna go downstairs, I'm gonna hook the MIDI drum set in and I'm gonna have elements on my page. Enlarge in size based on how hard I hit the drum.

• 40:39Scott Tolinski?

  That sounds like a ton of fun to make some of this stuff. So if you have any MIDI devices around, you can experiment with these. Otherwise, the audio synthesis API. I have this crappy little MIDI controller that's probably like 60 bucks. You can get them for super cheap. I got this, it's a simple little MIDI slider off of Etsy for like 20, 30 bucks. You can get all kinds of cool stuff out there to control MIDI and use that just for fun projects. Who cares? Build some art.

• 41:06Wes Bos?

  You can just make art.

• 41:08Scott Tolinski?

  Yeah, you're allowed to just make art.

• 41:13Wes Bos?

  Awesome. That was awesome. I can't believe how much of a well these APIs are sometimes, right? It's just one or two little APIs in the browser and it goes all the way down. And it has such history as well.

• 41:29Scott Tolinski?

  Yeah, totally, right? So cool. Let me know what you think. If you're building anything like this, let us know. I will make my synth available or my MIDI controller, my in-browser MIDI controller available. But just to let you know, it does not generate sound on its own. You need to pipe it into a digital audio workstation, at least right now. I don't feel like doing audio synthesis in it is what I want to use it for. Because whatever VST inside of Ableton or whatever I have is going to sound way better than anything that I could make myself with the Web Audio API. So either way, I'll make that available. We'll have a link in the show notes if you want to check it out. That code will be open source too, if you want to see exactly how I did it. As always, let's get into sick picks and shameless plugs, Wes.

• 42:12Scott Tolinski?

  What do you got for sick picks and shameless plugs?

• 42:15Wes Bos?

  Yeah, I'm going to sick pick a little microcontroller. This is the one that I used to program my Roomba. Actually, this is one that I'm playing with a little bit of these LED arrays right now. But these things are extremely fun to work on. They're a little microcontroller that has Bluetooth and Wi-Fi built right into it. You can power them via USB, you can power them via several other ways. They're $5, which is unbelievable. There's lots of projects out there for home automation. There's lots of projects out there for LED. WLED is one that runs on it.

• 43:00Wes Bos?

  And it's just, it's a really fun thing. Like, if you're trying to like tinker and get into electronics, for like, probably for like $10. $5 for the controller, $5 for some wires and whatnot. You can start building some, some pretty fun stuff with it.

• 43:18Scott Tolinski?

  I'm very interested in this WLED. Are you doing stuff with like, pushing visuals to an LED panel?

• 43:25Wes Bos?

  That's what I'm working on right now. So I've used WLED for the mirror in our gym. So when I built the mirror in our gym, I put LEDs behind the entire thing. And then it does, it can do like cool effects and like kind of backlight the mirror. And then my on-air.

• 43:44Scott Tolinski?

  Mm-hmm.

• 43:46Wes Bos?

  There we go. That is a sign that says on-air. And I just gutted it and put a ESP32 and some LED strips in it. And I can control that however I want. But now I'm, yeah, now I'm dipping into these LED arrays.

• 44:02Scott Tolinski?

  That's what I want, right there.

• 44:03Wes Bos?

  Which, I'm trying to figure, I'm like, how sick would that be if we built a website where you could just draw on the website and it would in real-time paint to LEDs?

• 44:13Scott Tolinski?

  Yeah.

• 44:13Wes Bos?

  So that's my idea, at least. There, it's a lot of LEDs, though. So like, the powering them is, is a bit of a thing I gotta figure out yet. Because it's, it's not just like, it's, it's a lot of power that these things take. If you have them all at full cook, you need like, like a massive power supplier or several of them. So I gotta, gotta dip into that, yeah.

• 44:36Scott Tolinski?

  Wow. Yeah, that's interesting, man. I'm fascinated about that stuff. Let's uh, let me update, or give me an update on that once you're, you're cooking a little bit more. I'm super interested.

• 44:45Wes Bos?

  Yeah, you should just, just order some, order a couple of these and at least like get it running. So you can kind of get in the headspace of it. Cause it's, I bet you can probably get it up and running in like 23 minutes. You flash the microcontroller in the browser. Like you don't even have to download like an IDE or like install Python or any of that. You literally visit a website, plug it into USB and it flashes it from the browser, which is, it's using the web serial API. It's unbelievable.

• 45:17Scott Tolinski?

  Man, cool. Man, you can do the coolest stuff sometimes.

• 45:22Scott Tolinski?

  Awesome. So my sick pick is going to be, I have after, what? Uh, many, many years working in computing on computers, I have finally given in and gotten a mouse. That's right.

• 45:34Wes Bos?

  Alright.

• 45:34Scott Tolinski?

  I never worked with mice ever. I always, I'm a trackpad guy, you know, I'm super trackpatty.

• 45:40Wes Bos?

  We're Costco guys.

• 45:41Scott Tolinski?

  Yeah, we're Costco. But you know what? When I broke my finger, the um, the trackpad was giving me all kinds of RSI. I'm like, now's the time. I'm gonna get a mouse. I got the Logitech MX Master, the S3, 3S? 3S.

• 45:56Wes Bos?

  3S. Yeah, that's what I have as well.

• 45:58Scott Tolinski?

  And I gotta say, the coolest thing about this mouse, besides the fact that it's just nice and comfortable and all that, the software that Logitech made for this is crazy good.

• 46:08Wes Bos?

  Yeah, it's actually pretty good.

• 46:09Scott Tolinski?

  You think you gotta get into, you know, some third party to do some crazy stuff, but I didn't realize this Wes, you can set every single button on this thing to respond to gestures. So every single button you could hold and move the mouse in a direction and it would trigger a different piece of functionality. So I have like-

• 46:28Wes Bos?

  And do it per app as well.

• 46:30Scott Tolinski?

  Per app as well. I, I have it so like if I hold essentially the, there's the back and forward buttons are typically what they are. I have my forward button. If I hold that and hit to the right, it does paste. If I hit to the left, it does copy. And I've been just copying and pasting that way and it feels so natural. It's crazy how natural it feels. Once you get used to all of like your own predefined setup. So yeah, I'm impressed. I'm impressed with the mouse, but I'm equally impressed with the software that this thing came with. So uh, shout out to Logitech for making something that's- Actually extendable and like customizable out of the box. If you're a tinkerer, this thing's a tinkerer's dream. I just have been going nuts with customizing this thing. So I know I'm super late.

• 47:12Wes Bos?

  Cool stuff.

• 47:13Scott Tolinski?

  Yeah. Very late to the party.

• 47:15Wes Bos?

  They were at GitHub Universe. They're like, like half going after like the gamer market, but like as soon as I ordered this thing, every single gamer, I did a TikTok on it and every single gamer comes out of the woodwork being like, oh, it's not fast enough for refresh rate. And like, all this, like, I find it lags quite a bit when I'm killing noobs in my game. And it's, I'm not gaming with this thing. I'm clicking on buttons and, and like highlighting text. And it's, it's a totally different world when you're like targeting like web developers who don't necessarily, they care more about ergonomics and productivity than they do like refresh rate or something like that.

• 47:56Scott Tolinski?

  Neat stuff. Logitech. Shout out. Shameless plugs, head on over to YouTube. CJ released a crazy good video on the most starred projects in JavaScript in 24. Man, what a banger of a video. So check that out. We'll link it available in the description of the show. It's really fascinating because it's not just like, here's a list of the top 15 most starred projects. It actually breaks down like why. It gets into what they did and why they had the year they had, or why people cared about it this specifically in the last year. And it gives you a lot of context to where we are today in 2025. So well worth your time. I learned a lot from it. So shout out to CJ for putting that together.

• 48:41Wes Bos?

  Sweet. It was a very good video. All right. Thanks everybody so much for tuning in and we will catch you later. Peace.