861: Local Data: Sqlite, LocalStorage, Session, Cookies and IndexDB

Scott and Wes dive into the world of local data storage, breaking down the pros and cons of Sqlite, LocalStorage, SessionStorage, Cookies, and IndexedDB. They cover real-world use cases like user settings, offline data, and auth tokens, while sharing their favorite tools and strategies for keeping your data fast and secure. Show Notes 00:00 Welcome to Syntax! 00:30 Brought to you by Sentry.io. 01:43 Why store data locally. 01:55 User preferences and settings. 02:50 Not logged in state (shopping carts, etc). 03:30 Data for faster loading. 03:51 Privacy concerns. 04:25 Large files or drafts. 05:50 Auth tokens. 07:08 Where to store data. 07:11 Cookies. 07:48 Local storage. 09:15 Session storage. 10:35 IndexedDB. 12:15 BYOJS Storage. 13:41 SQlite via WASM. 14:12 Penalties of SQLite in browser via WASM. 15:29 PGLite. 16:23 Dealing with migrations. 16:55 The advantages of the approach. 18:42 Dexie. 19:59 Patch messages. 21:25 A few options. TinyBase Docs. Local First Web. Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott: X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Scott Tolinski?

  Welcome to Syntax on this Monday, Hasty Treat. We're going to be talking about local data. Locally, right now, Wes, I'm trying to get a snack in. I don't know if you've ever had these Rice Mochi Balls from Trader Joe's, but holy cow, they're my favorite thing in the whole world. It takes great restraint for me not to eat them all.

• 0:18Wes Bos?

  Eat the whole bag, yeah.

• 0:20Scott Tolinski?

  Yeah, I don't keep them locally in my office usually because then I'll need to access them all the time. But if you need to access your data locally, we're going to be talking about that in this episode. But what happens if you need to access your data locally and that data's not there and you're not checking to see if it's there correctly? Well, you're going to get a bug in your site. And oftentimes, you know, with this stuff, especially if we're talking about syncing engines, making sure the data is where it's supposed to be, you can often get in these situations where we think we're in the happy path all the time. And oftentimes, we might not even realize what that happy path is. I experience that all the time on my Habit Path app because I use local data and data syncing. And man, it is hard to solve those bugs, but not with Sentry because Sentry lets

• 1:03Scott Tolinski?

  me know who has what issues at any given point in my application. And I know what's happening beyond the happy path. It works for me. All my data is local. It works.

• 1:13Wes Bos?

  Nothing but sad paths in your Sentry. Just a sad place to be.

• 1:19Scott Tolinski?

  You know what? The best part is too, you know, Courtney also uses Habit Path. And just like most things, when other people use your stuff, it's really nice for them to be like, this is broken. And then I'll go check the Sentry, see what's going on, fix the problems. Head on over to Sentry.io/synthesis. Sign up, get two months for free with the coupon code TASTYTREAT. Sentry saves my bacon all the time. Wes, what's up, my man? How's it going?

• 1:43Wes Bos?

  Hey, let's talk about storing data locally. There's lots of ways you can store data in the browser. But let's talk about why you might want to store data in the browser first, right? Probably the first one is user preferences and user settings. You don't necessarily always have an account for somebody to store that in, or it doesn't make sense to always send that data back to the database every single time that it changes, right? So one example I'll have is on my course platform, as people are watching the videos, every I forget how long every, every time that video current time updates,

• 2:25Wes Bos?

  which can be every two to three seconds, I'm putting that data in local storage and same, same with playback rate, volume level, like all those settings, right? I'll stick that in local storage and then periodically they'll, they'll get synced to, to the backend, but it's nice to stick that in local storage or in the browser when you need it immediately.

• 2:50Scott Tolinski?

  Yeah. Another thing too, often that like, I think you see this a lot more native apps, which is something that people might be doing on the web more and more these days is letting you use the thing before signing up for an account or even maybe paying for data syncing. That's like a thing where people pay to sync the data and they say, oh, actually I need to have this available on all my devices. You can store that data. You can let the, the people use the application even beside, before signing up for an account or even choosing to sync it to a server somewhere. And that's just like straight up usage of your application. If it is like a to do app, you're physically saving all the mutations and doing everything in the local data. And in that same regard, you might be saving that data locally

• 3:35Scott Tolinski?

  from the server into local storage or into your index DB or a local storage or literally anything. So that way you can access it faster because if the data is already on the client, you don't need to go to the server to get it. So it's a way of accessing faster data as well.

• 3:51Wes Bos?

  Sometimes it can, it can even just be a privacy thing.

• 3:54Scott Tolinski?

  Yeah.

• 3:54Wes Bos?

  So I built a background removal tool. And the thing with people's photos is they don't want to upload them to some random sketchy service that they're, they're going to have, right? So this background removal tool will just do it entirely in the browser. And the thing about that is if you refresh the page, they're gone. So you can stick those files and stick all the data related to that in the browser in, in one of the places we're going to be talking about. And that way, when you come back and refresh the page or, or even like, I was on JS bench the other day, writing some benchmark tests for JavaScript. And I like refresh the page before I had like saved it. Or TypeScript Playground is another good example.

• 4:39Wes Bos?

  Before I've saved it, right before it's in their database, if they even save that in their database, then it's gone. So it's nice to just persist it locally in the browser before you even have decided to send that to a server.

• 4:53Scott Tolinski?

  Yeah, I think anytime, like you said, anytime that you would expect your application to, if you hit the refresh button for to remain in the same state as when you left off. You're filling out a form. You're shopping at a store. You're you're you have a draft of a tweet, those types of things, right? Anytime you want that state to persist is a perfect opportunity to save your users some headaches by putting that that information in local data. Let me tell you, there is nothing more frustrating than filling out a big ass form and having the page refresh for some reason and having to start over. If you are doing job application things where they ask you to fill out every single one of your experiences or any of that stuff,

• 5:36Scott Tolinski?

  you gotta have that data storing locally or something so that way the user isn't losing that. That drives me absolutely crazy as a user. Yeah. And your users will delight be delighted for it, I'll tell you that. Another one is auth tokens. You know, we don't often think about this, but anytime you log into anything, it's either storing it in a cookie or local storage typically, whether it's a session token or a JWT as an access token, those types of things. These are stored locally on your client. Reason being is anytime you have a request to that server, you need that information there at the initial request. That way the server can be able to authenticate it as you go or authorize it.

• 6:17Scott Tolinski?

  Man, I don't know about you, but I flip flop authorization and authentication so much. I'm talking with the ZeroSync folks all the time and I'll use the word auth authentication and they'll be like, you mean authorization, right? I'm like, yes, I do. Gosh darn it again. Flip flops in my brain for some reason.

• 6:34Wes Bos?

  I would use the word authorization. Authentication is where I'm at. Authorization is like a like a credit card thing to me.

• 6:43Scott Tolinski?

  No, no, no, they're they're they're very distinct in web development or even any sort of login system, right? And I'm going to mess this up. Authentication is.

• 6:55Scott Tolinski?

  Logging in, right? It's the process of logging in and authorization is the process of making sure you have access to the things you have access to.

• 7:02Wes Bos?

  Okay, that makes sense.

• 7:03Scott Tolinski?

  Yeah, totally. I always just goof it up, though, when I'm trying to say it quickly. Let's talk about where you can store your data.

• 7:10Wes Bos?

  All right. So first one is cookies. Storing data in your user's browser, you can throw them in cookies. There are HTTP cookies, which can only be set and read via the server and server requests. And then there are just regular cookies, which can be set and read via JavaScript as well. Cookies have a limit to how much data you can stick in them. So generally, cookies are only used for things like language settings, session IDs, just like identifiers and short little text settings for that specific user.

• 7:48Scott Tolinski?

  Yes. Next up is local storage. Local storage is often more used for single values like a JWT, which could then be encoded, decoded, those types of things, or a simple string value. Or you might use a local storage for preferences, anything that you need to just save. Again, typically just a single string, save it, retrieve it, those types of deals.

• 8:09Wes Bos?

  Yeah, local search is, you can fit a lot in there, but the thing about local storage is that it's only text, right? It's key value store. So if you are trying to store something that is not text, you have to convert that to text. So if it's an object, you JSON stringify it. If it's an image, you can store images in local storage by converting to base 64. It's probably not the best bet, but it's a really simple API where you can simply just say localstoragesett item, localstoragethatgetitem, you can pull it out. The only thing that gets me sometimes is I'll Pull something out of local storage and assume it's an object and assume you can JSON parse

• 8:51Wes Bos?

  it, but it's not, not always the case, right? Sometimes you accidentally set undefined and then try to JSON parse undefined and you're in trouble.

• 9:01Scott Tolinski?

  I'm almost always saving things in cookies over local storage for no particular reason other than I just do. But nowadays, if I'm storing data like beyond single values or whatever, I'm using indexed DB typically. There's also session storage. So session storage, just like local storage, however, if you close the browser, you close the tab, that's the end of the session and it goes away. I've never used session storage. I'm going to be straight up about that. I just haven't hit a situation where I needed session storage, but I'm sure there's plenty of reasons to use it.

• 9:37Wes Bos?

  I think a good example of session storage would be, you were just talking about like you're filling out a form or a job application, you want to be able to refresh the page and that data to persist. Session storage will persist in the same tab if you refresh the page. It'll not if you open up a secondary tab. And that's really important if you, somebody is potentially on a shared computer. So you imagine a job application, somebody goes to apply for a job, they fill out the thing, maybe they don't finish it, and then they close the tab. If you put that in local storage, somebody else comes around and, and opens up the same website, then there's a bit of a privacy issue because now you're sharing

• 10:19Wes Bos?

  your, your data between users of the same device. Whereas session storage would, as long as they close the tab, you will, you'll be fine. It's cleaned up.

• 10:31Scott Tolinski?

  Yeah, that makes way too much sense. So thank you for that. Next up is indexed DB, which like a lot of people, I think, they say that it's hard to work with. And honestly, I've used it just, you know, raw, straight up API and I didn't find it to be that awful. It's not like the most fun API. It's not as simple as just get, set, whatever. However, I do use Dexy quite a bit. Dexy is a small library on top of IndexDB. And it's basically just giving you a nice little API for creating, getting, updating data. And you have to do a little bit of setup, but it's more like working with things that you're typical,

• 11:12Scott Tolinski?

  like normal databases that you work with. And for the most part, I found Dexy to be a nice to work in. And pretty much anytime I need to store structured data, an object of data, I'm not picking cookies or local storage. I'm picking IndexDB, obviously, because that's what it's made for.

• 11:30Wes Bos?

  You can't query local storage, right?

• 11:32Scott Tolinski?

  Right.

• 11:32Wes Bos?

  You can simply just check if a key is there or not, whereas IndexDB is a database and you can query it and filter and do all the stuff you're used to. And I picked up DexyJS for that background removal project I did, both to store files, which the browser also has file storage. We did a whole episode on that, but you can also just store like blobs of data in IndexDB.

• 11:57Scott Tolinski?

  Yeah.

• 11:58Wes Bos?

  And the cool thing about it is it has like React hooks and Svelte hooks. So that as long as you save it to IndexDB, your application is reactive and it's your whole state store. And it's a really cool project. I'm a big fan of it.

• 12:14Scott Tolinski?

  I want to introduce you to a library from Kyle Simpson of You Don't Know JavaScript fame. He has a library, he has like a kind of a bunch of new packages called BYOJS that are all kind of cool. And one of these is a storage package called storage, BYOJS.dev/storage. And the stated goal of this project is to have a simple key value storage API for all of the storage mechanisms. So you have a basic get and set and the only difference is, is where you import those from. So you import it from IDB for IndexDB, from local storage, your session storage cookie, OPFS, which is the private file system.

• 12:58Scott Tolinski?

  And so, yeah, basically, it just gives you the same API to work with and all of this stuff. And I thought this was a really neat API that you don't really hear a lot of people talking about.

• 13:08Wes Bos?

  That's cool. It seems like it's only key value, though, right?

• 13:11Scott Tolinski?

  It's only key value.

• 13:13Wes Bos?

  To query. You know what I really like about some key value stores? Dino's key value does this is you can set an array of keys. So you could say like, user and user 123. And that way you'd be able to query all the users or just query user 123.

• 13:31Scott Tolinski?

  Yeah.

• 13:32Wes Bos?

  I'm sure more key value databases are like that.

• 13:35Scott Tolinski?

  Sure.

• 13:36Wes Bos?

  But Dino's key value store is one. It's built on top of SQLite.

• 13:41Scott Tolinski?

  Oh, there's also a host of really wild new options that people are exploring in this area right now. As we start to get like more down this rabbit hole of using local data more frequently in applications, people are wondering, what can we do that is beyond index DB? So a couple of these things are using Wasm. So one of them is SQLite via Wasm is increasingly more of an interesting option. Although, you know, who said, I believe there is like a penalty, essentially like an initial download penalty, where the first time you you load it up, there is a pretty large hit to use something like this.

• 14:24Wes Bos?

  Yeah, you have to download. So the way what it's doing is it's literally running SQLite in the browser. So you have to download the entire binary to run SQLite via Wasm. And I'm curious how large that actually is. Oh, holy smokes.

• 14:45Scott Tolinski?

  410KB?

• 14:46Wes Bos?

  Guess how big?

• 14:47Scott Tolinski?

  410 kilobytes.

• 14:49Wes Bos?

  Uh, yeah, 558 kilobytes, but 1.14 megs if you want the async version. I'm not sure what that means, but that's much smaller than I would have thought to be able to run SQLite in the browser via WebAssembly. I think there's also like a

• 15:11Wes Bos?

  Going between the browser and WebAssembly, there's like a cost you pay for going across that bridge. But we had Johanna Schickling on, and he's one of the devs behind WebAssembly SQLite, and he knows the stuff. So I kinda, kinda trust him with this.

• 15:29Scott Tolinski?

  Yeah. There's also another one, PG Lite, which is Postgres in Wesam. And that's being created by the folks over at Electric Sequel, which is one of these cool new local first platforms. And I think Drizzle even has support for PG Lite now. Yeah. So, man.

• 15:50Wes Bos?

  That's what I want. I don't want to learn IndexedDB.

• 15:53Scott Tolinski?

  Oh, come on.

• 15:54Wes Bos?

  I already know. Well, I like, I can learn it. But you know what I mean? Like, I want to be able to just use the same database queries everywhere.

• 16:03Scott Tolinski?

  Totally.

• 16:04Wes Bos?

  And especially even if you're sharing schemas.

• 16:06Scott Tolinski?

  Yes. And schemas. Drizzle does have, they even have a whole page which we'll link to, which is Drizzle and PG Lite, how you can use the two of them together, again, to do your Drizzle DB Selects and all that stuff directly from PG Lite. Which, you know, the big question I have with some of these still is that, because I haven't used any of these more intense ones, is migrations and how do you handle that kind of stuff in a client side level? I guess you're running the migrations on boot up or something? Yeah?

• 16:38Wes Bos?

  Yeah, yeah, probably. I bet it's, you try to keep them a little bit more flat. So I guess you don't have to. But yeah, that's a great question. You'd have to run it in the, in the user's browser, because that's, that's where it exists.

• 16:54Scott Tolinski?

  Yes. So, that's all great, right? You can store data in any of those numbers of ways. So, you know, and primarily when you're doing this stuff, especially in applications that are loading this data, you can think of a lot of the times that this data, since it's living on the client, is a use case for not waiting for a long server response. Because if the data is living on the client, right, you don't necessarily- necessarily always want to wait for the server to hit the database, to do all the database stuff on the server before returning HTML. You want to get the client-side HTML and JavaScript as fast as physically possible. Then the data is there. It doesn't take any time to read it from the local database.

• 17:37Scott Tolinski?

  It's infinitesimally small to load from a local database compared to loading from a database on a server and building the HTML. So you're doing a lot more client-side rendering, which is one of the reasons why I've been talking about client-side rendering more. It's that you're sending the information to the client, the client is then grabbing the data, loading it up, getting you all ready to go. And then occasionally, if you need it, in the background, you could be fetching the data from the database. That database process on your server-side database could be kicked off right away. So that way, by the time that that comes back, you initially have data, you initially have a loading thing that's beyond a blank skeleton screen, and then the rest of the data can be filtered in in any sort of way.

• 18:21Scott Tolinski?

  Now, you could have like a little syncing message to let people know that data is coming in, or you can just pop it in, which might not be a great solution if you're getting your UI popping all over the place. Or you could have a button that says, you know, there's new data, refresh, those types of things. But in reality, I think what it comes down to is that, once you get into syncing data to a server, you're going to want to do it a couple of ways. Here's how I've done it specifically with storing things in Dexi. You store things in Dexi, and it's stored primarily in your client-side database. When you make a change, you push that change up to the server. When you load your application, again, you fire off a message that gets then pushed into your local database.

• 19:05Scott Tolinski?

  That database updates the UI, and then that information is in your local database. You're just trying to keep those two in sync. At a basic level, for an application that one person is using, you're sending a message off on one device, whatever, that's very simple. We talked about how it can get really complex with conflict resolution. But the way these things will work, I believe, in the future, specifically a platform like ZeroSync does this really well. It stores your information locally and it handles that sync process. And you're only ever writing one, one set of queries or you're writing one set of mutations all from the client side and it just knows what to do. It's sending messages back and forth.

• 19:49Wes Bos?

  You're not, you're not being like, oh, you're not writing the glue code yourself to sync it and query from the right place because that sounds like a, a really hard thing to, to do yourself.

• 19:59Scott Tolinski?

  Yes. And one of the cooler things that many of these new libraries are doing is that instead of like the naive way, I've done it the naive way a whole bunch of times, if you need to sync, by the way, there's plenty of application where you just save things locally and that's it. That, that's definitely super valid. So you don't always need a sync. The, the naive way is just to send all the information every single time. And that's too much information to send every single time. But if you're not dealing with a ton of data, not a huge deal. But the more and more data you add to the system, the larger and larger those sync messages are going to be. So a lot of the things that the, the new platforms are doing is via like patch messages. So it's only sending the data that's changed.

• 20:41Scott Tolinski?

  So it's asking for what's changed since this version. Oh, this has changed. That way that sync message becomes smaller every single, not every single time, but it's smaller than it would be if you were sending all the data, meaning that that trip becomes much less, meaning that you don't have that crazy pop in or anything like that because it's not sending a massive bundle of data that then needs to get updated into your local database. So that is how you get it done with local data. Local data, I think, is widely underused, but I think it could be, you know, greatly more used in terms of, like we mentioned, keeping persistent state for things like forms, surveys, things that would piss the user off if, if they lost their data in between save or refresh.

• 21:25Wes Bos?

  Can we rattle through a couple of the possible options for this? I know we did a whole show on it, but the author of TinyBase, when you were explaining it to me, I was like, I don't really understand where this is. And he said he explicitly added a diagram to the docs for us. And it's exactly what you were just explaining, right? Like, it's local, but you can also synchronize, like it's a local database-ish. But then it will also synchronize and it can also persist to like a Cloudflare or something like that.

• 22:02Scott Tolinski?

  Yes, and I believe James was on the LocalFirstFM, LocalFirst Podcast, and gave a really great explanation about how TinyBase came to be. So that's well worth your time to listen to. But yeah, TinyBase is an option here. Replica, I've talked about Replica before. It is tough. Replica is becoming zero sync, which in my opinion is way, way, way easier than Replica. PowerSync, man, PowerSync is one of these ones that handles the syncing. But every single time I look at this platform, again, I, just too many graphs and diagrams to the point where I'm like, all right, what are you doing for me here?

• 22:41Scott Tolinski?

  Yeah, it's tough. RXDB is a cool one as an option for, again, the ones that make this the easiest are the ones that kind of combine the two. InstantDB is a closed sourced Firebase competitor that does local storage by default. That's pretty amazing. Electric Sequel has kind of undergone some changes lately where they're, they're kind of like, they're changing somewhat significantly. I know it was like become version two now, and I haven't checked in on it since they announced that it's going to change a little bit. But Electric Sequel, like we mentioned earlier in this episode with the PG Lite, has definitely been one of the bigger innovators in this space, along with the Replica folks and many of these platforms, honestly TinyBase as well, for sure.

• 23:26Scott Tolinski?

  There are a lot of these options that we've talked about it here. Yeah, I'm going to paste in, LocalFirstWeb.dev has a whole section on different software. For these things. Yeah, I think there's a lot of different opportunity here. My, the most exciting one to me is zero because I've been, it's kind of the thing I've been like really hoping and wishing for.

• 23:46Wes Bos?

  Been hacking with, yeah.

• 23:47Scott Tolinski?

  And I've been using it for a little bit. Granted, there could be other ones like Instant DB is a really cool option, but it isn't like, I can't spin up an Instant DB server myself. And that's the only bummer about that, right? You're, you're tapping into their ecosystem to get going. But when I wrote an app with Instant DB, it was probably the closest thing I had to get zero to 100 in this space because it comes with like auth and magic link auth and all this stuff. Like you could have it up and running in no time. You're just not able to run it yourself at the moment.

• 24:19Wes Bos?

  Awesome. Well, that's local data. Hopefully you learned a thing or two. Let us know if you have anything to add. Tweet us, skeet us, email us, whatever you got. We're at syntax FM.

• 24:32Scott Tolinski?

  Yes.

• 24:33Wes Bos?

  That's it.

• 24:34Scott Tolinski?

  Yeah. And let us know what you're using local data for. If there's anything that we didn't mention in this episode that you think is a good use case for storing local data, leave a comment below, YouTube, Spotify, wherever you're at.

• 24:45Wes Bos?

  All right. Peace.

• 24:46Scott Tolinski?

  Peace.