921: AI Coding Roadmap for Newbies (And Skeptics)

Scott and Wes break down how to code with and for AI; perfect for skeptics, beginners, and curious devs. They cover everything from Ghost Text and CLI agents to building your own AI-powered apps with embeddings, function calling, and multi-model workflows. Show Notes 00:00 Welcome to Syntax! 03:56 How to interface with AI. 04:07 IDE Ghost Text. 05:45 IDE Chat, Agents. 08:00 CLI Agents. Claude Code. Open Code. Gemini. 11:13 MCP Servers. Context7 14:47 GUI apps. v0. Bolt.new. Lovable. Windsurf. 19:07 Existing Chat app like ChatGPT. 22:37 Building things WITH AI. 23:32 Prompting. 26:53 Streaming VS not streaming. 28:14 Embeddings and Rag. 31:09 MCP Server. CJ’s MCP Deep Dive. 32:36 Brought to you by Sentry.io. 33:25 Multi-model, multi-provider. 36:27 npm libs to use to code with AI. OpenAI SDK. AI SDK. Cloudflare Agents. Langchain. Local AI Tensorflow. Transformers.js. Huggingface. 44:12 Processes and exploring. Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott: X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Wes Bos?

  Welcome to Syntax. Today we've got an episode for you on an AI coding roadmap for anyone who maybe has has put it off for a while, or you're relatively new to coding with AI, like writing code with AI, as well as writing code for AI, meaning that you like use the libraries as part of your application. So we're saying for newbies, but we're also saying for skeptics where I know there's probably quite a few people, probably myself included in at to some degree, hopefully all of us to some degree, where you're kind of sitting there being like, hmm, is it really that good? Is it really going to change how

• 0:42Wes Bos?

  we how we do this type of thing? So we're gonna, we're gonna run through basically how to actually write code, all the different interfaces, and then also building things with AI. So like, understanding all of the different parts, context, tokens, prompting, all of the different models that are out there, streaming, embedding, rag, all those words that you're going to be hearing about. What's up? My name is Wes. I'm at the cottage right now, and I forgot my nice camera. So this is actually not, not bad, though. This is my iPhone 13, just on a tripod, and it's, that's decent video quality, eh?

• 1:19Scott Tolinski?

  Hey, man, it looks great. And yeah, happy you're out there at the cabin. That's a right time of year for doing that kind of thing, right?

• 1:27Wes Bos?

  So I'm, I'm up at the cottage right now, and we were just talking about it. And me and Scott were not be able to talk to each other, and the video was glitching out. And I was doing, I'm on Starlink, and I was getting like, like 100, 120 megabits down. I was getting about 10 to 12 up, which is, is usually not an issue. But it was just not working, right? And the pings were decent. So I switched to my LTE cellphone, and I'm getting four down, two up. And the video is way better. And the ping is, the ping is fine. So I don't know, man, that's

• 2:05Scott Tolinski?

  Yeah, it's bizarre. Yeah, definitely bizarre. And it definitely is easier to talk to you right now. Uh, hey Wes, I did some soldering, um, yesterday. No, this weekend, not yesterday, on Sunday. Well, I have this, I've been digitizing all my bands' old footage, which are on like Hi8 DV taped. I don't even know if it's DV. They're on Hi8 tapes. Which are these like tapes of size. And so I've been digitizing it all, throwing it on YouTube because like, hey, man, you know, that stuff's just gonna get lost to time if I don't. I have a little device that takes an RCA in and does HDMI out, uh, so that way I can do the cam link and whatever.

• 2:44Scott Tolinski?

  And the USB, micro USB came unsoldered from the board and I soldered it back on.

• 2:52Wes Bos?

  That's impressive. Wow. That, that's like very, pretty tiny.

• 2:56Scott Tolinski?

  It's, it's tiny. I'm doing the little thing. Listen, if I bump it the wrong way, it, it, the connection goes out. So I just gotta like, leave it. It's not, it's by no means a professional job, but I'm a, a terrible solderer. So I, I just wanted to, to, you know, toot my own horn here for a second.

• 3:15Wes Bos?

  Can we say that micro USB was the worst of all connectors?

• 3:19Scott Tolinski?

  It's got those little prongs in it. So every time you pull it in or out, it either stays in or like, uh, comes right out immediately.

• 3:26Wes Bos?

  Yeah.

• 3:26Scott Tolinski?

  Yeah.

• 3:27Wes Bos?

  Mini USB was far superior to micro USB.

• 3:31Scott Tolinski?

  In terms of shape or design or what?

• 3:34Wes Bos?

  I don't know. It just was better. It was easier, it's easier to figure out which, which way it went in. Micro USB sucked.

• 3:41Scott Tolinski?

  Yeah, it sucked for sure.

• 3:41Wes Bos?

  Probably it goes Lightning as the very lowest of all.

• 3:45Scott Tolinski?

  Yes.

• 3:45Wes Bos?

  Um, and then all the way up to like, you what, USB-C, although they got their own problems on top.

• 3:51Scott Tolinski?

  Lightning decided that a wafer was the best design of a vanilla wafer that could just snap off at any time.

• 3:58Wes Bos?

  All right, let's get on into it. First, we're going to talk about how to actually interface with AI when you're writing code. So the probably the most entry level one you maybe have experienced this is just like ghost text or autocomplete. So you fire up an IDE editor and whatever editor you're using. So obviously VS Code, Windsurf and Cursor, probably the big three right now, but any editor using will have some sort of AI plugin for it. And then as you're typing, it will try to autocomplete it. That can be a little bit frustrating at times, but I also find myself when it's turned off, I'm almost at a point now where I'm just like, or like you're writing code somewhere that it doesn't

• 4:42Wes Bos?

  have that, like a text area or, or like even like on Twitter and you're just like, I need that, you know, where, where's the autocomplete for it?

• 4:51Scott Tolinski?

  Yeah. And I will say that this is the lowest barrier to entry one. And honestly, probably the least intimidating version, because most of us are used to working with autocomplete of some regard inside of our text editor. And that ghost text popping in there shows you what it's going to be. You can choose to just toss it in there or not. So for me, this is like, if we're talking dollar signs in terms of intimidation factor, this is, this is like a $1 sign. It pops in there. You can choose to accept it or not. And I think that is a relatively easy way to get, get going with, working with AI and not have it not feel super icky. Right?

• 5:31Wes Bos?

  I know there's lots of people who don't even like autocomplete, like not even like AI aside, people, a lot of people don't like autocomplete at all. So if that's you, maybe you want to check out one of the ways to sort of chat with your code base. So all of these IDEs are going to have a sidebar where you can go ahead and select a model and be able to chat with it. Most of them now have defaulted to what's called agent mode. And what agent mode will do is you type into the box, add a component that does X, Y, and Z, and update the database. And then we'll go ahead and do it. But there are other modes. Where you it will either just give you the answer, which is like the very basic chat.

• 6:13Wes Bos?

  You can ask it to summarize something, you can ask it to modify something and it'll just give you the answer. You can go ahead and copy paste it, or it will allow you to apply the edits manually, which quite honestly is is kind of kind of slow. And a lot of people didn't use that this like chat feature until the agents came along, where it just sort of staged all of the changes. And then you can either you can go through every single change and accept that one by one, or you can just kind of YOLO accept everything.

• 6:43Scott Tolinski?

  Yeah, and the way a lot of these things, especially like cursor works now is that like, once it makes the changes, those changes are ready for you to use in the browser.

• 6:54Wes Bos?

  Before you accept.

• 6:55Scott Tolinski?

  Before you accept without without you having to commit to those changes, you can actually try that stuff out. You can see what it looks like, you can see what it feels like beyond just having to go through the code. You can see really quickly if it did the job by looking at the the end result and trying it out, which is one of the things I feel like I've had a harder time with some of the CLI agent platforms. I'm interested to get your take on that. But that's the one reason I tend to reach for the IDE chat first and foremost, as my main means of working with AI currently, specifically because I do like to be constantly trying it, constantly seeing it in action, and then go through the code and then choose to accept it or not.

• 7:38Wes Bos?

  The fact that you can always decline the changes or roll back your code to a certain point in time is really nice. It's a feature that I really like about all of these AI agents because you otherwise, if it tries to do too much, or you're a bit like spooked about how much it's done, then I have the same thing as well. Maybe we'll talk about that. So like the the third thing here is is CLI agent. So instead of asking cursor inside of the IDE to do something, the we're seeing a lot of people use terminal UIs to interact with your code base. So The big ones being probably by far the biggest one is Claude code.

• 8:21Wes Bos?

  And then a couple, as of recording, a couple of days ago, Google rolled out their own Gemini code. There is Codex from OpenAI. There's Open Code, which will allow you to bring your own thing. There's several others out there. We'll probably do a whole show on, on these CLI agents themselves. But the idea is, oh, and Warp, Warp, my favorite terminal, has like totally pivoted their business

• 8:47Scott Tolinski?

  Yeah.

• 8:47Wes Bos?

  into being like a coding tool via the terminal, which is like, I had a call with them like a couple of months ago. He's, he's like, I really think that like the, the terminal is the UI for, for building software. And I was like, really? And then like, here we are, a couple of months later, everybody's going bananas over terminal UI. So it's kind of interesting.

• 9:10Scott Tolinski?

  I still, as much as I use Claude code, I still

• 9:13Wes Bos?

  Yep.

• 9:14Scott Tolinski?

  don't buy it as being the best way to work compared to the IDE approach.

• 9:21Wes Bos?

  Yeah, I agree. And I think that they are different approaches. So the idea with something like Claude code is you, you type into the box, hey, build X, Y, and Z. And it will make a plan and it will go off and do, do that thing. And I think the idea with those is that you're not necessarily just sitting there waiting for it to complete. Is you're firing off three or four of them and they all work

• 9:44Scott Tolinski?

  Right.

• 9:45Wes Bos?

  in, in parallel. But for, for times when you do want to sit there and, and, and do it, I don't like that interface at all. If I hit the wrong key, the whole thing quit. Granted, the quality, I think the quality of Claude code is significantly better. I think that, I think that's why a lot of people like it. And you can also build Claude code into other tooling because it's, at the end of the day, it's just a NPM script that you can go, go ahead and run. But quite honestly, I much prefer the IDE experience where you have a full blown GUI and you can click on things, you can roll things back, you can accept things. And I know Claude code, you can do these things, you can open it up in an editor, you can view the desk, but it's not as good.

• 10:30Wes Bos?

  Not nearly, in my opinion, not nearly as good as, as using like a cursor via code in the editor.

• 10:37Scott Tolinski?

  If we're talking about skeptics and newbies too, I feel like they might have a harder time with this because especially if it's not getting it exactly right and you're constantly having to go back and forth with it, you you might be getting a little frustrated using the CLI, especially like, like I said, skeptics are going to be like, move, let me do it, you know?

• 10:57Wes Bos?

  Yeah.

• 10:58Scott Tolinski?

  And I think that's harder to do from the CLI than it is from the IDE itself.

• 11:04Wes Bos?

  I agree, I agree. So we've got ghost text agents, chat agents, background agents, right? CLI tooling that we can use. And the next, next way that you can interface with this is MCP servers. This one's not necessarily direct, but the idea with the MCP server is that they are, are kind of just like plugins for your AI and they will allow you to bring more context, more tools to the AI agent. So a big one that I found out about a couple of weeks ago was Contact 7. This one will give you the ability to inject the documentation for different libraries into your actual prompt. And you don't have to do it yourself.

• 11:47Wes Bos?

  You just say, go read the docs for Tail and For or Svelte 5 and then it will go off, download the docs, stick that into the prompt for you, which is, I honestly quite like that a lot better than the like cursor built in documentation. I've never felt that the cursor built in documentation worked very well. I'm not sure if you have thoughts on that.

• 12:10Scott Tolinski?

  It's obnoxious because in, so folks who don't know, newbies, skeptics, anybody, in Cursor, the editor, in the settings, you can attach docs and to like a keyword essentially, like Svelte 5 docs and you're pasting in either a link to the docs page or the llms.txt file, which we'll talk about. And with that, you are able to access the up to date documentation and reference it. However, with Cursor, you like frequently have to do at the docs in like every message. At least I have found to get specific syntax, I'm always having to reference constantly those docs to say, no.

• 12:51Wes Bos?

  Yeah.

• 12:52Scott Tolinski?

  Please, for real, actually use these docs these times. I'm begging you, I've given you the docs, you have the docs, you can look at them, you've indexed them, just use them, please.

• 13:04Wes Bos?

  And especially for docs that don't have like a single page that you can grab them. A lot of documentation now will will publish their documentation either as a single markdown file or like llms.txt or as like multiple markdown files, which you can easily gather up and inject as you need it. But I've had some documentation for very small libraries where it's like 30, 40 pages and I just hadn't found a good way to provide all of that. So I'm still still kind of working on that.

• 13:38Scott Tolinski?

  Yeah, my, my biggest frustrations do still revolve around versioning modern software. It's like if you are, there's a reason why it seems like we're converging on a handful of tools because AI is really good at a specific era of specific tools. And like, once you get into, oh, Svelte is now releasing, wow, they changed their on colon click to on click. And like, AI is just like, I've never heard of that, even though I have all the documentation in front of me. Yeah.

• 14:12Wes Bos?

  Anytime you find that you could, you say the AI is really dumb at X, Y, and Z. The AI, for example, AI can't do math. I talked about this in our MCP episode, right? If that is your like thing, oh, it sucks at X, Y, and Z. The answer to that is very likely not, oh, okay, well, scrap it. It's not very good. But the answer almost always is, yeah, there should be some sort of tool added to it via probably via MCP that will give it the, the powers that you need.

• 14:45Scott Tolinski?

  Yeah, totally. Word. There's also a handful of GUI apps, V0 probably being the most, most popular one. It's, it's basically a way to build things very quickly using Vercel, V0, well, not using Vercel specifically, but it's, it's built on Vercel. And

• 15:06Wes Bos?

  It's a chat. You should say that. It's basically like a visual chat that will

• 15:10Scott Tolinski?

  It's a visual chat.

• 15:11Wes Bos?

  Build out the UI and, and some of the functionality.

• 15:14Scott Tolinski?

  Yeah. And a handful of these tools all work very similarly. And the, the branding on a lot of them is really like, We can build something very easily for you. So another popular one would be lovable in this space. Lovable has pretty much the exact same landing pages V zero in many ways. Like you can tell just by looking at the landing pages, like kind of what the vibe is for some of these things. Bolt new, another one. Basically, hey, you want to build something? You want to start with a chat inside of a GUI? Then like, hey, this is the way to go. This is maybe in my mind, I know a lot of people like V zero, but in my mind, this is for more of the

• 15:58Scott Tolinski?

  person who doesn't want to get into the code as quickly themselves in terms of like in your, in your editor, in your, in your dev environment, like up and running. Like this is somebody who wants to build something with the least amount of effort, but also maybe the least amount of human, human interaction with it. Right off the bat.

• 16:20Wes Bos?

  And it's certainly a lot more UI focused than it is like, like all of these tools can do backend now. Initially, it was just UI focused, but it's much more based on the visual prompting and going back and forth, which is, is really cool, quite honestly. And also Windsurf has this feature built in as well, right? Because the type that want to just like have it generate code are more, more going towards the IDE. And then the type that wanted to just build a visual website are more going towards and like components and whatnot. They're going towards these tools. So Windsurf is trying to, at least I think they're trying to attack those tools as well.

• 17:01Wes Bos?

  So I wouldn't write these off because they don't seem to be as popular in our circles of people who already know how to code.

• 17:09Scott Tolinski?

  But that's what I'm saying.

• 17:10Wes Bos?

  I'm always, oh, and REPL, REPL is huge in this, in this one as well. And especially REPL, because it seems like nobody in my circles uses REPL. And it seems like REPL is absolutely huge, which tells me that there's like a whole generation of people who are building stuff are just sort of coming at it like they don't care. People that just want a website don't care if they're doing it the right way. They just want it to work and they want it to, they want it to build what they want, right? So those are pretty interesting things.

• 17:43Scott Tolinski?

  A lot of these tools are popular within the I built a app with AI and now it makes me five grand a month crowd Twitter thread boys down below type of audience. So just like you said, I should say, uh, more popular with maybe not our kind of deeply tech audience. And as far as like the AI skeptics go, might be most skeptical of this type of thing, just because of the type of individual that most.

• 18:13Wes Bos?

  Yeah, the brain dead. No.

• 18:19Wes Bos?

  Uh, no, I certainly very careful to write these off because I think that they are very good. And also, if you were to give me a tool like this for something that I was not good at, like, for example, making art, chat apps, you know, being able to generate photos, or I use, I use the 11 labs one to create sound effects for my, my captcha one, you know, and I'm sure the audio people are sitting there being like these idiots are just typing in what they want and they're getting these awful sound effects out the other end, but I don't care. I don't care. It gave me a really fun, you burned it audio, audio little clip and it worked great.

• 19:03Scott Tolinski?

  Yeah, I know. It's definitely great for that. Um, though the last one here is just like chat apps, like chat GPT. And I gotta say, I used AI to do coding like this for a long time because for a little while before you had things like before the, the chat interface was any good. Cause that for a while, the chat interface was really bad inside of both VS code and, you know, it made more sense to either do the autocomplete. But if you really wanted maybe like a back and forth with some code iterating on a code, copying and pasting your code into chat GPT and then working with it and then copying it back into your app for a while was like the best way to

• 19:44Scott Tolinski?

  work in AI before we, we got a lot of these newer tools. So this is certainly still something you can do if you're the type of person who's like. I really wanna dial in on this. Like, I wanna copy and pay, I wanna work back and forth with this in a chat way. But at the end of the day, it's not a text editor. You're not getting into there to modify the code. You're doing a lot of copying and pasting and trying a lot of back and forth. And it is really super inefficient compared to most of the newer styles of working on this stuff. So I probably wouldn't recommend this. If you're talking to me and you're new to this stuff, I would say you can either start with the ghost text if you're scared of it.

• 20:24Scott Tolinski?

  Um, but the IDE chat is probably the least scary way to start using AI in your app in small dosages, um, instead of like trying to go whole hog into this thing.

• 20:39Wes Bos?

  Yeah, I will, I will still use chat apps for research and reference where I don't want it to take my entire repo into consideration. Because sometimes you ask it questions and I'm like, no, I don't want your, I don't want your idea to be slanted based on the code I've written. I want you to just give me an idea of what you think in general. So if, if I'm looking up how CSS features work or how to best approach something, I'll, I'll paste it in. But I put this in here because at, at least maybe even three or four months ago, it seemed like a lot of people thought that coding with AI was, was that you're just copy pasting it into chat GPT and copy pasting it out and like, no, that's slow, that's slow as hell. And I, I wouldn't wanna do that either.

• 21:23Scott Tolinski?

  Yeah, for sure. And I gotta say, Wes, your, your note on research in those types of things. I have completely changed how I do that in the past month. With code and in general, I'm almost entirely using DIA Browser now for that specific thing. The chat window in DIA Browser is so good because you know how some documentation sites have like the chat with AI feature?

• 21:51Wes Bos?

  Yeah.

• 21:52Scott Tolinski?

  In there, DIA basically gives you that for any project. So you can reference the docs. In the browser from DIA with your code, you're pasting back and forth, you're referencing other tabs. Oh, I have this library and I have this library. They're in two different tabs. I can pop open a little chat on the one and say, hey, if I were to integrate it with this other service here in this other tab, how might I do that? Here's some of my code and it's like so stinking good for that. So DIA, man, has been, and I actually just signed up for the perplexity comment browser beta as well. So I think we're going to start seeing agentic browsers more and more.

• 22:33Wes Bos?

  Cool.

• 22:34Scott Tolinski?

  Yeah.

• 22:35Wes Bos?

  Yeah, I'll have to give that a shot. Um, let's move into the second portion of this episode, which is actually building stuff with AI. So this is not writing, getting AI to write code for you. Um, it's kind of the other way is around is, is you, or I guess the AI is writing the code. But I feel like in order for you to get good at writing code with AI, you need to actually build something with these tools in order to understand what all of the pieces are and how all they all fit together. And if you build like a small little application that has a few of the features, I think it will give you a good idea of like token counting, pricing, context, prompting, the different models that are

• 23:19Wes Bos?

  out there, streaming text versus not embedding what this word called rag is. So if you build something that includes all of these features, let's go each of these features one by one and sort of talk about them. One, first one is just like custom prompting, right? You want to build an application and you're going to send some data, most likely text, but it could be a multi multimedia. If you're going to be sending it to an AI and getting the data back, custom prompts is probably the, the most table stakes, right? When you build something, you, you give it an initial system prompt and you can say you are an

• 23:58Wes Bos?

  expert at identifying birds, given, uh, given the sounds that they make when they're scared. You know, some absolutely silly write that. And, and that will sort of set the stage for it. So a custom prompt is the very basics there.

• 24:15Scott Tolinski?

  Yeah. Have you solved the getting data back piece? Like I need the data back in this specific format. I understand that there's open AI functions and things like that.

• 24:26Wes Bos?

  Yeah, yeah, the being able to get data back in the format that you want has has pretty much been solved. Even like the syntax website, we have AI stuff on it. This was before you were able to the OpenAI SDK, it has a has the ability to just pass like a JSON flag to it and it will always return you JSON. If you're not using an OpenAI compatible provider, which we should say almost all of them are, you use the OpenAI SDK and you can use it with obviously OpenAI, but you can also use it with with Claude and Gemini and like pretty much all of the other services have been compatible with it, which is really

• 25:10Wes Bos?

  nice because you can always just swap out models and you don't have to like, you're not hard coding your app against using just OpenAI models. But yeah, it's that's largely solved. Generally, these AIs are best at returning either JSON or Markdown, but you can ask for it to return other formats as well. I think JSON and Markdown is is probably the best. Like Cloudflare changed their Wrangler format from TAML to JSON. And I kind of have a feeling that I wonder if partially the AI being able to modify and generate those things better in JSON versus TAML is part of the reason why they did that.

• 25:55Scott Tolinski?

  Yeah. And I've been seeing a lot of JSON-L as being the way that people have been working within LLMs with JSON. Have you been seeing that yourself?

• 26:05Wes Bos?

  What's JSON-L? I know I know JSON-C is with JSON with comments.

• 26:10Scott Tolinski?

  Yeah, JSON-L is basically line separated objects. So if you were to have

• 26:19Scott Tolinski?

  Just a JavaScript object on a line, you could have just another JavaScript on an object without

• 26:25Wes Bos?

  Okay.

• 26:26Scott Tolinski?

  Yeah. Each line represents a JSON object, ideal for handling large data sets, streaming data scenarios. JSONL's efficiency makes it popular for choice with fine tuning language models, so.

• 26:40Wes Bos?

  Oh, that makes sense because with JSON, you'd have to wait for it to return the entire thing before you can go ahead and parse it.

• 26:47Scott Tolinski?

  Exactly.

• 26:47Wes Bos?

  That's a pain in the butt. So JSONL will allow you to parse the data. So maybe we'll, we'll jump ahead to one of them, which is the idea of streaming. So these AI models are often very slow. If you want to simply send a request, sit there and wait for the entire thing to come back. So what you have to do is you have to start streaming the data as it's coming back from the AI. And if you have stuff in there that needs to be parsed, most likely Markdown or JSON, you want to be able to display it in some way. As it's coming in, you need to be able to obviously parse it. That's why sometimes when the, the result is coming in, like let's say you're streaming in a Markdown table.

• 27:31Wes Bos?

  For a split second, you might see like a weird half-formatted Markdown table, and that's because the whole table syntax hasn't come back. So with the idea of, with the idea of JSON lines is you get simply one object each time, and at least you can, you know, you can safely parse that entire line. Before you go ahead and, and use it.

• 27:52Scott Tolinski?

  Yeah, because that is something now that you, you actually have to worry about because, um, yeah, like, like you said, if, if you're waiting for that entire JSON object to come in. Or you're having to figure out where the lines are, where the new objects are, where the new rows are, whatever. That makes the process of streaming data in from just way more complex and obnoxious. So for sure.

• 28:15Wes Bos?

  Let's talk about embeddings and RAG. So this is something you'll hear quite a bit about. So when you want to prompt a model with custom information, a lot of people think that you would train a model on something. Like, let's say, for example, we have almost 1,000 syntax episodes, and you want to ask it a question about a specific, "In what episode did Scott do X, Y, and Z?" Or, "What are Scott's thoughts on Svelte 5?" The AI model doesn't know about Scott's thoughts on Svelte. Actually, it does, actually does know quite a bit now.

• 28:54Scott Tolinski?

  Well, it will also confidently know about it, even if it doesn't. It'll just give you hallucinations about it.

• 29:00Wes Bos?

  Yes. No, but what needs to happen is if you have custom data that is the AI model's not trained on, you have to then provide that data as part of the prompt. And we can't take the transcript of 1,000 episodes and stuff that into the prompt and then ask it what you want. So what you need to do is you need to basically search your data and then find the pieces of data that make sense. And then you give it to the AI model, say, "Given the following context, please answer the question, what does Scott think about Svelte 5?" And then you might give three or four different pieces of the transcript. And in order to find which of those pieces of

• 29:44Wes Bos?

  the transcripts you need to provide to the prompt, you need to first convert your data to embeddings. And text can be converted into embeddings, which is essentially just like a large vector array of data. And then you can find items that are closer to it. So what you do is you would convert your question to an embedding. That will return this array of numbers. And then you, maybe you convert each of your episode transcripts into embeddings, or you convert like three or four sentences at once into embeddings. You can get pretty complex with it. But essentially, what you do is you say, "I'm searching for this. These are the things that are,

• 30:27Wes Bos?

  these are like the pieces of the transcript that are close enough to it." Then you can take those three or four and then provide it to the prompt and get your answer. And the same thing works with images as well. That's how Google similar images work. So you might take an image, convert it to an embedding, and then you take every other image in the world and convert those to embeddings. And if if images have similar like items inside of it, the embeddings might be might close. Or if they're similar colors, they might be close to them as well. And that is really nice way to add additional context to that problem. That's called RAG, Retrieval Augmented Generation.

• 31:08Scott Tolinski?

  Yeah. All right. The next one is an MCP server. And I gotta say CJ just released an incredible video giving a deep explainer on actually building with MCP. We've done episodes on MCP here, some very recently as well. So MCP is largely a means of using AI to connect with other items via like an API. So like if you want to connect with something else via your chat or via your AI, it can use MCP as an interface to go off and do other work, whether that is interfacing even with like a full on other application or another service, another server somewhere.

• 31:51Scott Tolinski?

  So I highly recommend giving CJ's video a watch because nobody explains it quite like him in terms of like getting your hands dirty and really building something.

• 32:00Wes Bos?

  Even even Figma has a MCP server as well. If you want to get details about a component, I was watching Adam Wath and he was trying to figure out how to convert Figma components to Tailwind code like pixel perfectly. And he got pretty far using the Figma MCP server.

• 32:22Scott Tolinski?

  Yeah, yeah, I've been like I mentioned, I've been using MCP in Godot to change things in the Godot interface. There's MCP for things like DaVinci Resolve. I mean, there's a lot of different MCP out there for interfacing with different applications. The Sentry MCP is the is a great example. So Sentry as sentry.io is the perfect place to track all of your bugs to fix them. Seer is a new product from Sentry that allows you to debug your code and find out exactly why it sucks with AI. It's amazing.

• 32:53Wes Bos?

  Yeah, it's agentic. It figures it out for you.

• 32:57Scott Tolinski?

  It is crazy good, but there's also a sentry MCP server that again allows you to interact with your sentry and gain access to understanding your, your code base and errors in your entire, uh, everything that's happening. Cause sentry essentially is, is insight into how your application is functioning. Uh, you can gain so much information from interacting with the sentry MCP server. So check it out at sentry.io forward slash syntax.

• 33:25Wes Bos?

  Next up is, uh, multimodal. So if you want to build something, uh, probably text is, is the most common way you're going to be getting responses, but dip into, to the different models, right? There's models out there that will allow you to send an image as part of the prompt. There's models out there that will return images or video as part of the, part of the prompt. There's models out there that will transcribe audio, right? It'll do speech to text. There's models out there that will do text to speech. I did a really silly one, uh, probably a year, year and a half ago where it took the webcam photo of me took, um, and then took a photo and then it sent that off to open AI and said, describe this

• 34:08Wes Bos?

  image as if you were Ricky from trailer park boys. And then I took that response and I piped that into 11 labs, which was a custom model trained on Ricky from trailer parks voice. And then it played it back to me, right? And like, just like putting those two things together is just so much fun because you can obviously string these things together and then throw MCP into, into the mix. And before you know it, you got Ricky from trailer park boys, like fixing your sentry bugs or something hilarious like that.

• 34:40Scott Tolinski?

  Yes. And, and for those of you who might be listening to that and being like, what's the point of that? What's the point? The point is, is that you're building things that are at least getting you interested to have some sort of output to explore and to give these things a rip. So that way you're not learning about it in some dry context or something without, uh, you know, an idea in general. And you say, I wonder if I can do this and this and this. You start connecting these things and it's going to help you really figure out the whys and hows. So that way, when you want to build something, um, practical, you, you have an exact idea of how the pieces fit together.

• 35:19Wes Bos?

  Exactly. And like the multi models as well as you'll get a very good understanding of how much does this cost? You know, there's models out there that are fast and cheap, but not very good. There's models out there that are slow and expensive, but very good. And trying them all out will give you a much better feeling for the trade offs on that for at least two weeks before they're all deprecated and new ones come out and you have no idea that three is better than four and everything has an O in it and you're all confused.

• 35:55Scott Tolinski?

  I love that.

• 35:56Wes Bos?

  But yeah, just man, also like dipping into like local AI stuff as well. We've got a note coming up with just like try out transformers JS, try out TensorFlow. If you are able to run them locally, you don't have to pay for them. They're they're pretty dirt cheap to run by communicating with APIs. But if you want to do it absolutely free, you want to try some really neat stuff just locally or doing it entirely in the browser. That's kind of a fun way to do it as well.

• 36:26Scott Tolinski?

  Yeah, totally.

• 36:27Wes Bos?

  Let's talk real quick about some libs and that you can use to code the AI. We talked, we said the open AI SDK is probably your, your, one of your first interactions with talking back and forth with these different models out there. The Vercel AI SDK is really, really nice. So this is a TypeScript toolkit for building, it's kind of like if you want to build a chat app or you want to like build common stuff and be able to interact with all these different AI providers. There's just a whole bunch of handy tools in there. So if you want to go ahead and build a little chat app, you want to work with streaming

• 37:08Wes Bos?

  and then also you want to integrate it with like React or Vue or Svelte. It's an awesome little toolkit that you need to grab. It's not, a lot of people think it's since it's from Vercel, you have to post it with Vercel and have to use React and NextJS. That is, that is not the case here.

• 37:26Scott Tolinski?

  Yeah, it is basically just a library for interacting with the AI SDKs themselves. It's kind of like a middleman. So that way you don't have to use the OpenAI library and the Anthropic library, or do just fetch calls for everything, right?

• 37:41Wes Bos?

  Yeah. And I think it goes a little bit further than that as well. Like it will give you UI components for, that will help you streaming and it's really nice. Like Vercel is really good at building SDKs. So I highly recommend you check that out.

• 37:56Scott Tolinski?

  Yes. There's also Cloudflare Agents, which I have not used yet. This is one, there's, folks, if you're feeling overwhelmed, my job is to use this stuff. I've picked up so many of these tools and I still cannot find time to use all of them.

• 38:10Wes Bos?

  Yeah, the Cloudflare Agent stuff is, it's funny, a lot of the Cloudflare examples use the Vercel AI SDK, which should tell you that it's not just for posting on Vercel if Cloudflare is using it. Yeah, the Cloudflare Agents one is a little SDK that will allow you to persist state if you want to like store stuff in a database, or you want to have long running tasks, if you want to query the web, you want to have synchronous workflows. They provide a whole bunch of tools on there as well. So the, the Vercel AI SDK, well, that's more of like a, something for interacting with the providers and providing UI. The Agents

• 38:53Wes Bos?

  SDK is more of a infrastructure level thing. So if you want to schedule tasks and persist state and tell something to run, like a, like a cron job every single time, it hooks into the Cloudflare infrastructure, all of their tooling that they have. So.

• 39:11Wes Bos?

  There they have tons of examples on their GitHub as well. If you want to dip into that, but I would probably tell you if you're just trying to build something here, you don't have to use all of these. I would probably just start with the OpenAI SDK. Just start with some calls back and forth in a no JS script. And then as you say, hmm, this is kind of annoying that I have to do this a thousand times. I wish there was something that made it a little bit easier. Then check out the AI SDK, the Cloudflare Agents. And then there's, there's also like LangChain as well, which will, have you used LangChain in the past there, Scott? It's been about a year for me.

• 39:46Scott Tolinski?

  I haven't used it, especially since, well, I've used it in the past, but I have not used it since their website has undergone a major glow up. I guess you could say it's very corporate looking now.

• 39:57Wes Bos?

  So LangChain allows you to chain together different AI stuff. So you're able to like, like ingest a whole bunch of documents and then send them off to one thing. So you end up just like clicking together a lot of these things.

• 40:13Wes Bos?

  When you're, you're building an AI application and LangChain is a toolkit for, for doing that as well. I haven't found my, there's a, there's a JavaScript version and a Python version, which is really nice, but I haven't found myself needing that in quite a while, when I feel like I can get away with just using the, the previous libraries we talked about.

• 40:35Scott Tolinski?

  That's what I'm saying too. Yeah. And there's also Transformers JS. It's basically allows you to interact with Hugging Face, which Hugging Face is one of those. I don't know how you feel about Hugging Face. I love, I love it. It is intimidating. This is one of those things to me that if I was a newbie or a skeptic coming into AI and I were to go to Hugging Face, I would feel intimidated by it because there's so much stuff and there's so much good stuff on here. And it does a great job of giving you demos and examples and how to use these things. But in the same regard, it's a lot of words and all of a sudden you're feeling like, wow, there's a ton of stuff in here that I might not understand.

• 41:20Scott Tolinski?

  How so many things exist here, how so many people are using this. But basically it allows you to use all kinds of AI stuff from.

• 41:31Scott Tolinski?

  Different models to different data sets. You can integrate these things in your application with Transformers JS, or you can just bounce around hugging face and give some of them a try to see if they are the type of thing you want to use.

• 41:46Wes Bos?

  Yeah, so Transformers JS allows you to use local AI models.

• 41:53Scott Tolinski?

  Yes.

• 41:53Wes Bos?

  In JavaScript runtimes, meaning that if you're in Node JS, you're in Dino, you're in the browser, you want to be running an AI model because these these big models like OpenAI, they're good, they're very good at lots of stuff. But sometimes that is too expensive or too large to do it. And you want a very specific model that's like very good at doing one thing.

• 42:18Scott Tolinski?

  So. Or you just want to be running it locally, you know?

• 42:22Wes Bos?

  Yeah, or you or you just want to be able to run it locally because you don't you don't want to have to send that off to the cloud, right? So they do like natural language processing, vision, depth estimation, image segmentation. You know, like if you want to be able to.

• 42:37Scott Tolinski?

  Painting, yeah.

• 42:38Wes Bos?

  If you want to, yeah, if you wanted to build something like Photoshop where you you draw over something and then it tries to fill it in, you could reach for one of their computer vision models in there. Or like I've done it in the past where you can try to filter out bad messages from people, you know? There's toxicity detection in there. The whole AI world is so deep on very specific models that are for specific tasks. And it's absolutely nuts how deep all of this goes. So certainly check out, I'm I'm such a huge fan of Transformers JS. I've built so many things with it, even just like smile detection.

• 43:20Scott Tolinski?

  Right.

• 43:20Wes Bos?

  If we wanted to feed it, I built something that it took, I don't know, the 6,000 screenshots from an image and fed it through it and and tried to figure out which one had the largest smile in or which one had the most scared face in and filter it on. Like that would cost a lot of money to send 3,000 images off. And that would be extremely slow as well to have to upload 3,000 high resolution screenshots to an external service.

• 43:52Scott Tolinski?

  I will say that if you're the type of person who wants to build fun things and, you know, explore a little bit, Transformers, JS, and via Hugging Face, is a great way to do so. I think that, like you mentioned, all those things that you said are a great way to get your hands dirty with some interesting things. So I guess that leads us to some process things we've mentioned throughout this episode. But like, I think one of the biggest things that I can suggest to people is to explore. You know, we mentioned that there's all these like fun and interesting things. I do think that if you're the type of person sitting on the sidelines and saying, "I don't need that stuff",

• 44:34Scott Tolinski?

  I think you need to explore your curiosity a little bit. Because we are all about making cool things, we're about exploring technology, we're about building things, and like, if you leave the word AI out of it, and I were to tell you as a developer that you could build something that detected a smile and took a screenshot when there was a smile, or I could build something that, you know, did this or that really fascinating thing, like generated an image or something like that. If you leave the entire big AI scary cloud out of it for some people.

• 45:14Wes Bos?

  Just call it machine learning. It sounds so much more scientific and sick.

• 45:18Scott Tolinski?

  Is that not exciting? That is exciting that you can build that kind of stuff. And like, you would have told me five years ago, 10 years ago, that I'd be able to build some of this stuff or do some of this stuff. I would have been like, "That is incredible. There's so many neat things I want to do with these ideas." So, explore. Don't be afraid to go out there, explore. If it breaks, if it doesn't work, toss it out. I have a directory on my computer called temp inside of my sites folder. It is used for a scratchpad websites. I throw up together demos. Some of them work, some of them don't. The cool thing about my temp folder is it automatically deletes itself. So because of that, like, I can just toss stuff in there and if I ever

• 46:03Scott Tolinski?

  actually like the process that's happening, I can move it into a real project or whatever. Otherwise... It's just scratch. It's just exploration. And I think that is an important feature that I think some of us lose because we, we really like, we take into the this, I don't know if it's like the social context or like, you know, we latch on to these feelings.

• 46:27Wes Bos?

  There's a lot of awful stuff around it, you know, like there's

• 46:29Scott Tolinski?

  Yes.

• 46:30Wes Bos?

  The, they're stealing content. It uses a ton of energy, you know, the whales are dying. Um, there's, there's a lot of awful stuff around AI and I, I, I don't want to discount that, but I do think that a lot, there's a lot of benefits of, of it as well. And I do think that this industry is certainly changing quite a bit. So if you're a bit of a skeptic, just know that

• 46:57Wes Bos?

  we understand, but this is, this is kind of exciting stuff and you can build some really neat stuff. I I, especially if you're like at all, a little bit burned out. Build some fun stuff. So build some

• 47:09Scott Tolinski?

  Build some fun stuff.

• 47:10Wes Bos?

  Silly Pokemon indexer thing and you can go ahead with it. Turn off the like people generating the like dogs that are chat, are playing poker or whatever. Just turn off all that garbage and, and build something cool.

• 47:25Scott Tolinski?

  Or do that. If that, if that's what scratches your edge, if that's what scratches the ear behind the edge, uh, it's behind the ear, then do the dogs thing. That's fine too. Also, another tip I have is if you're getting started with these things, especially for the first or second time, or you failed at them in the past, use tech that you already know because you can tell if it's generating code, like if you're working with, with AI tooling, it, you can tell if it's generating code that you want to actually use because you know the code, you know what good code looks like. And therefore if you're choosing to accept or bring in code into your code base from AI,

• 48:09Scott Tolinski?

  you're going to know a lot better and faster if it's, it's code you actually want to have, if it's code that you know and understand. So use tech that you already know, if possible here.

• 48:20Wes Bos?

  All right. That's all we have for you today. We got no sick picks for you today because we are recording ahead for the summary and I am, I am out. I'm flat out of sick picks. So

• 48:29Scott Tolinski?

  We can only buy so many things here. Uh, we're going to go broke trying to find sick picks.

• 48:35Wes Bos?

  No sick pick, but down in the comments below, let us know your sick pick. I'd love to, to hear it. And is that, that's all you got today, Scott? That's all I got. All right. Thanks for tuning in. We will catch you later. Peace. Peace.