Scaling Time-Series Workloads on Postgres

Many real-world processes produce data as a continuous stream rather than as isolated records. Sensor readings, financial markets, and application telemetry all generate data this way. This kind of time-series data has a distinctive shape. It’s written far more often than it is updated, it accumulates continuously, and it is usually queried across ranges of time. Time-series databases are built specifically for this kind of workload. TimescaleDB https://www.tigerdata.com/timescaledb is an open source database from Tiger Data https://www.tigerdata.com/ that adds time-series capabilities to PostgreSQL. It’s implemented as a Postgres extension, so it introduces new functionality while preserving standard Postgres behavior and SQL. This lets a single Postgres-based system handle both transactional and analytical workloads without splitting data across multiple tools. Brandon Purcell https://www.linkedin.com/in/bpurcell/ is the Director of Product Management at Tiger Data. In this episode, Brandon joins Kevin Ball to discuss why time-series data breaks conventional databases, how hypertables and Hypercore scale Postgres, zero-copy database forking for agent-based workflows, and much more. Full Disclosure: This episode is sponsored by Tiger Data Sponsorship inquiries: [email protected] mailto:[email protected] The post Scaling Time-Series Workloads on Postgres https://softwareengineeringdaily.com/podcasts/scaling-time-series-workloads-on-postgres/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  Many real-world processes produce data as a continuous stream rather than as isolated records. Sensor readings, financial markets, and application telemetry all generate data this way. This kind of time series data has a distinctive shape. It's written far more often than it is updated, it accumulates continuously, and it is usually queried across ranges of time. Time series databases are built specifically for this kind of workload. Timescale DB is an open-source database from Tiger Data that adds time series capabilities to Postgres SQL. It's implemented as a Postgres extension, so it introduces new functionality while preserving

• 0:43Speaker 00

  standard Postgres behavior and SQL. This lets a single Postgres-based system handle both transactional and analytical workloads without splitting data across multiple tools. Brandon Purcell is the Director of Product Management at Tiger Data. In this episode, Brandon joins Kevin Ball to discuss why time series data breaks conventional databases, how hypertables and hypercore scale Postgres, zero copy database forking for agent-based workflows, and much more. Kevin Ball, or K-Ball, is the Vice President of Engineering at Mentos and an independent coach for engineers and engineering leaders.

• 1:24Speaker 00

  He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action Discussion Group through Laten Space. Check out the show notes to follow K-Ball on Twitter or LinkedIn, or visit his website, kball.llc.

• 1:51Speaker 01

  Brandon, welcome to the show.

• 1:52Speaker 02

  Yeah, thanks for having me, Kevin.

• 1:54Speaker 01

  Yeah, I'm excited to learn a little bit more about what you're up to. So let's start with you. Can you give us a quick overview of your background and then how you ended up at Tiger and what you're doing there?

• 2:04Speaker 02

  Yeah, no, absolutely. I've been working in the tech industry for a very long time. Spent my early years in the military, even writing code, and then started working at Adobe soon after that, working in a number of different roles there. And met a couple of guys when we built a startup. Startup was called Spatial Key, so the background in my area and experience in Postgres goes back almost 20 years. We built a geospatial analytics platform that was built on top of Postgres, primarily initially focusing across a broad range horizontal, but then switching that into insurance and everything from underwriting to event response and portfolio management type use cases.

• 2:45Speaker 02

  And probably after about 13 years, we sold the company and I spent some time, took some time off afterwards, and decided to come back and actually take a very different role. I was doing most of the leadership, I was the CTO of the company, and really wanted to get back into the tech side of it and product. And I love Postgres, and so ended up coming over to Tiger Data because I believe they have a great product and it really takes Postgres to the next level.

• 3:07Speaker 01

  That's a good intro. So can you tell us a little bit about the product at Tiger and what is it that it handles that Vanilla Postgres doesn't?

• 3:15Speaker 02

  No, absolutely. And so, just kind of taking a step back, you have Tiger Data as the company and the organization. And for most that are familiar with Timescale, which is the product, Timescale DB that Tiger Data has built, it's an extension of Postgres. And so by an extension, it's not a fork, it's actually an extension, and so it's offered as an open-source product. And the challenge is, a lot of times, when you have repeated time series type data readings, you end up growing that data pretty extensively. And oftentimes, as that grows in Vanilla Postgres, you start to run into challenges. And those challenges yield themselves as read and ingest, or ingest bottlenecks, read start to slow down as your data starts to bloat, and you run into challenges with indexes

• 4:00Speaker 02

  growing with a single table, time range queries get challenged. And so, Timescale DB addresses that by building this extension. And we'll talk about the number of the technologies that sit below the surface that address that. But from a developer perspective, it's the same SQL, no differences at all, but there's some intricate workings of the database that makes it easier to use and actually scale a lot better. And so from an offering perspective, we have a couple of different offerings within that, but most people start with the open-source product, and we also have our Tiger Cloud solution as well.

• 4:29Speaker 01

  Yeah, so let's talk a little bit about time series. And what are the, like, problem domains and use cases? Like, if you're someone who's coming from a web background, maybe you've only ever dealt with relational, classic rest objects and things like that. Like, what are the types of problems that get you into saying, "Hey, actually, I need a different data structure here. I need something that looks like a time series."

• 4:49Speaker 02

  People typically do start with Postgres. I think 60% of the world, you start with a relational database, Postgres being the one that's probably used the most by developers. But you end up having a particular device or set of devices that are emitting readings, and that can be across a number of different areas as well. Manufacturing or robotics, medical devices, finance and crypto are also used there, where you have a number of different transactions, smart buildings and industrial systems. So it's a number of different areas, and I think it's a growing area as well, as we look at what we call the physical world, where you have solar, energy, any number of things that are all emitting readings on a periodic basis. And the number of devices within that is growing as well. So you end up with hundreds of thousands or even millions of devices.

• 5:34Speaker 02

  We have one customer that has 22 million devices, and they're all emitting readings every few seconds, every second, in some even sub-second as well. So the volume of data that you're ingesting and the volume that's growing over time gets larger and larger, right? And so that challenges the databases in new and unique ways. The other piece of that is, is that you still need the transactional-relational part of the database as well. And so as you start to grow, and oftentimes people start with Postgres, and as you start to outgrow it, your query slows down, your ingest slows down, you have two options really, right? You can separate it off into another solution, so you bifurcate with some type of analytical database, and then you retain your transactional database.

• 6:18Speaker 02

  Or you can adopt a system like Timescale DB that gives you the best of both, where you have co-located into the same system, we can extend it, and this is with zero application changes at all. And we can talk about what that migration path looks like, but you're not having to make application changes, where if you bifurcate it, you have to manage multiple data pipelines between this other system. And typically, most of these systems have some evolution of SQL that is a little bit different, so you end up having to rewrite portions of your application.

• 6:48Speaker 01

  Yeah, that makes sense. So, I mean, I guess, looking at that piece, as an application developer, presumably the underlying data model looks a little bit different when you start to go to time scale. Is there any difference at all, I need to know, in terms of...

• 7:00Speaker 02

  Zero, zero difference at all. And so you think about, you're starting with a table, you have a reading, you have a timestamp, and then you have any number of attributes that are associated with that device. So that stays exactly the same as you move into time scale. What happens under the covers is you have a concept of, everyone understands tables in existing databases, but there's something called a hypertable, which takes that same concept, and it starts with the partitioning, the automatic partitioning of that data. So breaking based upon those time steps and partitioning it in chunks that separate it over time. So you're not reading all of data at once, and there's a number of pieces that are around that.

• 7:41Speaker 02

  And so within those chunks as well, we have something called hypercore, which allows us to have a hybrid row and column store. So typical database in Postgres, you have a row store, and your data is organized in rows, just like traditional databases, but below that as well, and this happens through policies and automatically, and developers have control over this as well. Data has migrated into a column store as well, and this is getting the value of compression. You know, we've seen customers...

• 8:09Speaker 01

  Right, yeah, it lets you be sparse, essentially, yeah.

• 8:11Speaker 02

  And that column store is, as we've evolved it over time, it used to be very immutable. And as we've moved and we improved that, it's very immutable as well. So the data can change within there as well. And so you get the benefit of the space savings. Typically, we see users with 90%, we've seen as high as 98%, where you have really duplicative readings oftentimes. And then you also get the value of vectorized query acceleration as well. So that allows the underlying processor through a feature called CMD to execute multiple batches against the column store at a time. And the result is you think compression is slower, right? But when you're compressing the data combined with a vectorization,

• 8:53Speaker 02

  we've seen acceleration in queries 10 to 100 to 1,000 times. And then beyond that a bit, we also, you think of normal B-tree indexes that you have in a row or store. We have something called sparse indexes as well, which are like min max and unblame filter indexes that allow for execution of queries to pull from the metadata that is in these batches or in these chunks. So you're not reading all the way down to the row level and pulling data. So common sum, query, first, last, things that you would have in this world would be much, much faster because they're reading from metadata versus from the underlying rows and having to decompress it. And so the combination of those things from a developer perspective is,

• 9:36Speaker 02

  you know, you migrate data over, there's a couple of commands you run to turn a normal table into a hyper table or create a hyper table from scratch. But when you insert data or when data is migrated, it automatically, based upon those timestamps, partitions it in the appropriate chunk. And that partitioning extends beyond just time series. We have numeric partitioning as well. And also like UID V7, which has a time based component as well. We also have support for that as well. So your data ends up being broken up into these chunks. But when you're querying it from a top level, it appears as one. And oftentimes, one of the biggest values of that is when you have billions and billions of rows that maybe go over 10 plus years. If you're querying more near term data,

• 10:21Speaker 02

  the query planner is only going to hit the chunks that are necessary for that. And so it's reducing that footprint that you're querying from.

• 10:29Speaker 01

  Yeah, no, it's interesting. This is bringing me back in more recent years have dealt with smaller scale data. But yeah, a decade ago was dealing with trying to, at that point, it was hundreds of millions of rows was where we started to have to think about all these different things and we're planning out partitioned tables and stuff. So drawing back on that, asking some of the questions I ran into then to see how you handle it, like one of the big questions there is around, you know, is this all time partition versus do you have other partitions? You know, is this non time? So I'm guessing from the name of this, this is all focused on time bucket based on day, month, year, or something along that? Or are there ways to utilize this underlying infrastructure for non time based partitioning?

• 11:08Speaker 02

  You can use additional partitioning schemes that come with Postgres. But for the most part, I find they're not necessary. And in the specific problem domain, people are typically querying by time as one attribute. And then if you're looking at in device world, by device or other data as well. And there's a concept that we use internally called segment by for how you segment the data into these batches, usually by a device ID, some type of tenant ID, user ID. And that allows the engine to group the data intelligently so that those batches compress better and also are more queryable and faster when you're querying the data. And that's a big part of that row store, column store hybrid approach where we're grouping those and creating those logical batches.

• 11:54Speaker 01

  Yeah, super interesting. Now, another thing that at that time was challenging was, yeah, anytime you, if you wanted to join from this over into something else, because at that point I was dealing with partition logic at the app layer. You're down in Postgres. Can I just interact with other tables and joins and things like that? How I might expect?

• 12:11Speaker 02

  Yeah, it works exactly. So think of it as from a developer perspective, really no different than what your traditional experience would be in Postgres or any other relational database. And users typically have metadata that's associated with these devices. Let's say, for example, or users that are stored in traditional normal tables that aren't partitioned by time. And you can join directly against those tables as well. And that's a pretty common need and use case where if you're going to bifurcate that and have it across multiple systems, you have to do that at the application layer versus the database layer. And so you have to query it and then unify that back at the application layer where you can just write a query, join it across the two,

• 12:54Speaker 02

  aggregate the values you want, and then display that back to a user.

• 12:57Speaker 01

  That's fascinating because yeah, I remember that being a huge pain point once we got to the place we were having to partition is suddenly all these things that we used to be able to do at the SQL layer, we had to pull into the application. Is there any secret sauce to how you do that and keep it performant? Because under the hood, you've got all these different partitioned things. Like, do you plug in directly into the query engine? Like, what does that look like to make that work?

• 13:19Speaker 02

  It's intercepting those queries and the query planner is actually executing. And this is part of the extension. And I think really the, the difference in the value of Postgres with its extensibility. I used PostGIS for years and I've been amazed at what you could do, accomplish from a GIS perspective. And now in Timescale DB is using those same concepts with the extensibility of Postgres and the ability to interact from a query planner, automatically route to the appropriate request against the right chunks. And we've spent a lot of time optimizing that path as well. So you touch as few chunks as possible for your queries. And so you can have a lot of historical data. And under the covers, these chunks are really just tables.

• 14:01Speaker 02

  And then there's a layer of metadata that Timescale is being able to use to organize that. And because of that too, your indexes, if you're talking about in the row store within those chunks are broken up across these those as well, which is one of the big challenges when you have a huge table, that index grows very substantially. And then you have an insert and has to update that index. And so in this case, and one of the tunable parameters that users have to figure out oftentimes, and we're making this easier and easier, is what is the appropriate chunk size, right? Like, we come out of the box with seven days. But if you have a really high volume data, you can maybe go down to even an hour. Right? And so there's some different optimizations techniques around that. But for the most part, it's pretty, from a developer perspective,

• 14:43Speaker 02

  it's pretty automatic and simple to get started and configured and get it right out of the box.

• 14:48Speaker 01

  That's interesting. So I'd love to kind of understand the underpinnings that you're giving those handles to. In particular, so one angle into this, you mentioned people often start looking at this when they run into challenges at the ingest time. And then when we were talking about performance optimizations, you talked about query speed, which is read time. So like, how does this help on the ingest time? And like, if I'm running into ingestion problems, like, does changing that bucket size help? Are there things I can tinker with what type of index I'm doing? Because I know, yeah, definitely in vanilla Postgres, some indexes are better for write heavy applications versus not. What are the knobs and levers here?

• 15:22Speaker 02

  The same things really apply to Postgres. The difference really comes down to, I think, the chunk size as well. So you end up with the active, let's say, chunk that you're inserting. If you're just append only type workload, you want to think about the size that that chunk is going to be as a row store before it's converted into the column store and compressed. And so the same kind of rules apply, but it allows you, because you have many chunks, you only have to worry about that one in terms of the indexes and that. And so what we've seen from that is as you have a normal Postgres table and you ingest over time, you see this linear degradation of actually the amount of rows per second that you can ingest.

• 16:08Speaker 02

  But with Timescale DB, if you're tuning things appropriately and you have the right chunk size as you work through that, you're not going to see a drop off in that at all. And on ingest, the routing algorithm for how it goes to the appropriate chunk. So it allows you to backfill old historic data as well. And that's going to be routed to old chunks as well. And by default, it goes into the row store and then it's converted for a policy into the column store. We do have the ability to even to write directly to the column store. And that's something we're evolving over time. And what we found is you can on nominal hardware, get to a million rows per second plus from an ingestion standpoint, which is very difficult to do in vanilla Postgres. And that's going to drop off pretty quickly as your data starts to grow.

• 16:54Speaker 01

  Yeah, for sure. So would it be fair to say, like, in vanilla Postgres, you see this linear degradation with the size of the table? In Timescale DB, is that essentially the size of the chunk?

• 17:04Speaker 02

  It's pretty much flat, I've found, is what you can see. And I think, I forget it was a Microsoft study or another one that did an actual independent test and benchmark of it. And we see that line is fairly flat. And the chunk size does matter more for probably query performance than it does for ingest performance. Because most of your chunks, unless you set like 30 days or two months for one and you have a ton of data, it's not going to grow substantial like you would see tens of millions or hundreds of millions in a single one. You're going to start to spread those out. And so you don't run into those same problems that you would see in normal Postgres.

• 17:39Speaker 01

  Makes sense. So another thing that I start thinking about when I think about time series style data is aggregates, right? I want to look at things aggregated by day or by week, or maybe many different versions of that. And if I'm coming from a vanilla Postgres world, maybe I have some materialized views that are computing that for me, or I have a stored procedure that's happening, or some application logic that's running this and dumping it into a table.

• 18:02Speaker 02

  Yeah.

• 18:02Speaker 01

  How does that work in the time scale DB world?

• 18:06Speaker 02

  So taking the same concept of materialized views, which are commonly used concept in relational databases, we've built an extension of that called continuous aggregates, which behind the scenes really are a materialized view. It's creating an additional hyper table, but it's aggregating data based upon, we have a function called time bucket. And I always forget the equivalent in Postgres. It's basically bucketing based upon a time range. And those are self updating as well, based upon policies as users you set for how often they're updated, how far back you want to update that. That allows you to have an aggregate level view of your data that is automatically

• 18:47Speaker 02

  self updating when any changes to the hyper table occur. And users will even often take those and build those hierarchical. So they build off of one another. So let's say you had a device that was emitting every minute and you wanted to roll up that was five minute, one hour, one day, one week. And you could have four different aggregates that build on each other that are scheduled automatically that update. And that allows you to have, will take what could be hundreds of billions of rows, and then breaking that down into individual time buckets and get sub low millisecond query times to be able to query against them. And we have customers that are

• 19:27Speaker 02

  in our Tiger Cloud environment that are have 20, 30,000 queries a second that are hitting billions of queries a day against what I would consider nominal hardware, not crazy specialized hardware.

• 19:40Speaker 01

  Yeah. Well, that maybe is a good opportunity to start talking about Tiger Cloud, right? Because everything we've talked about to date, I could get open source. I put a plugin in my running Postgres. And as you highlight, like, I don't need a very fancy hardware to run this. Why would I go to Tiger Cloud? What do you all add there?

• 19:55Speaker 02

  Yeah. So Tiger Cloud has a couple of different features around it. One is just the overall managed service, the ability to take the pain off the team to run a managed service, much like you would have with RDS on AWS, which I used pretty extensively before coming to Tiger Data for our geospatial product. But within that as well, there's a couple of other features beyond that. So the ability to have read replicas, HA replicas, all self managed by their team, having a connection load balancer that sits on the outside. So if you need to scale up, if a single instance isn't going to suffice across many different instances, the ability to scale across those many different read replicas as well. One of the other features I think that's worth calling out that makes a

• 20:38Speaker 02

  big difference for users that want to retain a long history of data without keeping all of that data hot is tiered data. And what it does behind the scenes is it takes a table, and let's say you have 10 years of history, and you want the first two years to be in hot queryable, and then you want the other eight years to be tiered. And that tiers that off to S3 in the AWS case, and then ensures that in iceberg format behind the scenes. But the entire time range is dual queryable as one table. So when you query the table, if you do the first two, if you're expanding outside of that first two years, it's actually hitting and combining those results together to

• 21:20Speaker 02

  allow you to query that full 10 years. Now, if you think about, if you had to do that manually, you'd have to have some archiving scheme and data pipeline that would go and move that data around. And there's a lot of that. Here you set a tiering policy, and it automatically starts tiering that data based upon that date. So you can have a much longer history of retention for longer query periods.

• 21:43Speaker 01

  That's fascinating. Well, and that gets into, I mean, there's all sorts of different things that people do right now in terms of piping data into warehousing, or analytic stores, or lake houses, or all these different things. Like, are you moving in that space as well? Or is this staying constrained to these like time scale bucket?

• 22:01Speaker 02

  No, it's, we're also moving in that space with a product that is due for full release later this year. It's still in beta right now, which is Tiger Lake. And that's to allow that bidirectional connectivity with other teams often where you would have that data, let's in iceberg format, pushing data to Snowflake, pushing data to Databricks, and ability to use those other processes. And like I said, that's bidirectional. Maybe we are bringing data in where Tiger Data or time scale becomes your operational database for your dashboards, where you need really fast responsiveness. Maybe you have a data science team or another analytical team that's consuming that using Snowflake, Databricks, or Athena in AWS, and the ability to have that live in both

• 22:42Speaker 02

  locations. And I've even seen it too, for compliance reasons, organizations need multiple copies of that data in multiple places. And so instead of having to build that data pipeline themselves, like kind of make that automatic where you just enable this feature, and it automatically, where based on the data that you've selected, it automatically starts writing using Tiger Lake out to S3. And we also have the equivalent on Azure as well.

• 23:08Speaker 01

  Yeah, super interesting. So I'm going to go a little bit more hypothetical now, because we're talking about these different things. And I'm like, starting to see the picture that this looks like, right? It's a lot easier to manage than, I mean, my current situation, right? I've got stuff in Postgres and a few different places, and then we're shifting things into AWS and have this whole processing pipeline. How do you see this changing the way that people manage their data?

• 23:30Speaker 02

  Yeah, no, I think the goal is is to make it easier. So people aren't focusing on the plumbing and all the work around it, and are focusing on the business problems and solving them and enabling really multiple teams to be effective using some of the same data. And the intent is is you can get that right, depending on where your source of record is, or where you're ingesting, if you're bringing it directly into Postgres, into Tiger Data, and then being able to use that downstream and other processes. Maybe a team writes their data directly to S3, and then that did pulled in and operationally enabled through Tiger Data. So it allows for flexibility for how you want to work and gesture data and bring it in, but then making it exposed and sort

• 24:11Speaker 02

  of democratizing that for the team broadly, but not having to worry about all the plumbing and all the work that needs to be done, which AI makes it easier from a development perspective. But if you don't have to manage to do that yourself, you're focusing on using the data and leveraging it both in your applications and your other downstream analytical processes.

• 24:29Speaker 01

  Yeah. So we talked about some of the plumbing that makes this possible, right? You have these the hyper tables and the hybrid row column stuff. Are there any missing primitives? Are there primitives that you all are still working on that you feel like are needed to realize that vision?

• 24:43Speaker 02

  And by primitives, I mean, I mean, we really support everything that's in Postgres today, right? So it is vanilla Postgres with these extension to be able to then partition the data. So if there are other, you mean from a data sync perspective or other areas?

• 24:59Speaker 01

  I'm just thinking, like, we're talking about a future that, I mean, as someone who increasingly is like, I just want to see the answers, right? I don't want to think about the details. Like, I really resonate with your vision for like, we're focused on the business problem. Let's focus there. Hyper tables, time scale DB was arising out of a thing that was a gap, right? There's a gap already. Do you feel like with this, like, all the pieces are there and it's about execution? Or are there like core underlying things that you all are working on that are like, oh, actually, we also need, I don't know, maybe it's like, I keep going back to non-time related partitions, because that's what I was dealing with back when I was dealing with the scope of data. But like, are there things on the roadmap where you're like, oh, this is really going to open another door?

• 25:39Speaker 02

  So I think it's more around enabling others to use these different things and some of the tooling around this to make it easier to consume and get set up. There are some pieces of the learning curve, and we've spent a lot of time building out our CLI and MCP, as well as something we call PGAI guide, which is a guide that will, you know, guide coding agents to leverage all of the capabilities and tools very easily. And so I've even personally just building some side projects, building a home project for all my data telemetry for my entire smart home, and then using like MQTT protocol to push that in, is I built a majority of that just giving it a description of what I wanted in the coding agent, using the CLI and MCP, and didn't even

• 26:23Speaker 02

  have to touch anything on the database side. And it built everything from the schema, the designs, the hyper table design, the continuous aggregate design, all of those things, very well guided to get you set up in just a few hours, right? And so up and running, and then tying that back into my home assistant, and then a couple other things on top of that. But being able to do that very quickly and enabling someone without having to, you know, you understand Postgres, and it's good to understand the concepts. But I think as we get, you know, more and more down this road, you want to focus on building and not necessarily understanding every technical concept. But from a roadmap perspective, it's making these things easier

• 27:04Speaker 02

  and more turnkey and ability to scale higher and higher, because the needs of what we're seeing as we focus as a business in the physical world, all of these industrial applications, it's a growing from data centers to, you know, solar, any number of things. The number of devices and the amount of data being emitted is just exploding and the volume of it. And so, one, making that easier, and two, making that easier to scale. So you have that long history of data without having to sacrifice the value of something like Postgres and the developer kind of productivity around it.

• 27:37Speaker 01

  Yeah, that's really interesting. It reminds me of, yeah, conversations I've had with people doing like test infrastructure for physical world stuff, like, the volume of data is unreal to somebody coming from a web background.

• 27:49Speaker 02

  Yeah, it's very different. And you know, because I built a lot of web applications in my time. And you know, you basically have crud operations with reading and writing. And it's a different problem space when you're you have something that's emitting a lot of time series data and it's an entirely, you know, set of different challenges. And oftentimes, people will look at an LLM or Google it and their specific problem, whether it's write ingest challenges, read problems or queries, and stumble upon, you know, something like time scale DB. And I'll be honest, before I joined, I didn't know it existed as a product. So to me, it was a new product. And so it was cool to see how they took the extensibility of it and built this in, you know,

• 28:31Speaker 02

  this specialized time series database without sacrificing all of the other capabilities.

• 28:36Speaker 01

  Yeah, so that's kind of interesting. So from that educational standpoint, we talked about some use cases earlier, but like, if I'm a developer, what are like the things that are going to cause me to like where I should be like, oh, that I should check out time scale DB? Like, what are the problem spaces that maybe I would ask my agent about? And like, I should just have my head in the space of, oh, this is probably a time scale or a time series problem.

• 28:58Speaker 02

  We see a lot of greenfield applications as the sort of the visibility and kind of people are more familiar with it. You know, I think anytime, because you're starting with Postgres, so you have all your relational stuff. But if you're building an application where you have any time series data and you know, over time, it's going to grow, I don't think you can go wrong in starting with it. And whether that's even local and building, you know, locally, it take you five minutes to stand up a Docker image and run it local and test it out. So it's very straightforward with a single line curl install. But if you know you're going to have data that's going to be, you know, emit a number of time series values, it makes sense just to start with that. On the flip side, if you're talking about brownfield applications and customers are starting to run into this issue,

• 29:43Speaker 02

  you know, typically, you're going to just have a significant volume of data and maybe you don't even have any, you know, query challenges or ingest issues yet. But just the footprint, and we've seen this, I did a webinar with CERN a couple of weeks ago. And, you know, they were using a previous database before and have migrated over to time scale DB. And one of the things they called out, they had like 70 terabytes of data for time series across, this is like the Hadron Collider and all the other devices in CERN. And by moving over and getting the value of compression, they reduced that footprint from 70 terabytes down to five terabytes. They were able to then actually run that on higher speed SSDs and VMES SSDs.

• 30:25Speaker 02

  So they just got better performance by default on all of their data because that 70 terabyte footprint that was spread out on over maybe more disk, they were able to get on faster disk as well. So the compression and reduction in footprint allows you to have, you know, using more commoditized hardware and get better performance, both in terms of throughput with multiple users hitting it as well as sort of, you know, and as you have any ingest challenges, if you have high volume of data as well.

• 30:54Speaker 01

  It's interesting, again, to think of like, oh, the data problems that CERN has, right? Like, they're like doing ridiculous amounts of just stuff coming off of every one of those runs. And they've, they've got to deal with that. Pretty different than those of us building web apps. I guess a lot of work now moving into the AI space as well. Like, it feels like there's so many more data points that are, we're trying to integrate tracing and observability and things like that. Are there any features aimed at, you know, agentic use of these tools? You mentioned like kind of exposing it for as a, in like the developer ecosystem. But what about other uses?

• 31:29Speaker 02

  This is one of the nice things as well. You don't have to have a separate vector database. You have, you know, PG vector, and we also have an extension PG vector scale that scales out to as you get to larger data sets. But you have your vector embeddings right alongside all of your other data that could be a part of this hyper table and time series data. So you have the value of a time series database with a transactional relational and, you know, the ability to have vector embeddings stored right alongside of that. And maybe that's what you were getting to in terms of the other primitives as well. You know, that's a common use case where you need that. And we also have authored something called PG text search, which is an open source extension for text search.

• 32:10Speaker 02

  And we've seen it, you know, scale alongside what you would see comparable to elastic search. So you have, and you can build a true hybrid search using a combination of text search and vector embeddings and then combining those two together. So we've seen it in like the telemetry space where oftentimes you have other metadata or their textual descriptions that you would want to search against to then get back to the underlying readings and the values of that. And so you have all of those technologies combined into one database where you'd often have to maybe split that over a vector database, a time series database, and then your relational, and then have to join that app application layer across all of them. You get that in one place. And then on top of that, the value of compression.

• 32:54Speaker 02

  So you get the reduction in footprint from a data standpoint.

• 32:58Speaker 01

  Yeah, it's interesting. I feel like every time I interview someone from a database company, one of my questions is, all right, why you and not Postgres? And your answer is, yes, why not? Let's do it with Postgres.

• 33:09Speaker 02

  Yeah, I know. That's why I, it's one of the reasons I joined too, because I just have loved Postgres just because of the extensibility of it. And one is an open source product. And then the community is amazing in terms of how many other extensions, you know, because you can install and have those other extensions right alongside. So you're not limited, you know, and we talk about extending primitives and other areas. You can just, you know, add those other extensions in.

• 33:32Speaker 01

  Yeah, that makes sense. Well, let's talk a little bit about what's coming down the pipe. What do you have on the roadmap? What should we be watching out for as we look at, you know, Tiger data and time scale DB and all these things?

• 33:45Speaker 02

  I think it's continuing to build on the, you know, the Tiger like story with connectors into Snowflake, connectors into Databricks, you know, longer term as well. Those are areas we'd like to continue to expand on. And from a developer perspective, our goal is to make this sort of out of the box, not having to do, you know, the tuning exercises, making continuous aggregates easier, you know, making the developer experience really world class. And that's from both the database and getting set up, but also our tooling, the CLI and MCP, those other tools around it, not only for setup and configuration. But one of the questions we often get from developers is how do I know I'm set up correctly? Validation, being able to validate that it's configured properly.

• 34:29Speaker 02

  And with guidance based upon our SA team that that's helping guide customers to be able to scale that as well. So one of the other areas is the, you know, the common agentic workflows as well, where you have agents working against databases. We're internally working on a evolution to our storage engine called Fluid Storage, which is sort of a zero copy type storage that allows you to create forks without the under, without having to copy the underlying data. And then it will be for any changes, it will diff from there as well. And combined with our, like our CLI and MCP, the ability to, you know, in your CICD process or your agentic processes, quickly fork those databases off until versions of it.

• 35:14Speaker 02

  So you can take your production database in, you know, seconds or minutes, be able to fork that and then have an agent that can work against that data and then stand that up and tear it back down very easily. And so you may have different flows from an agentic standpoint that just allow you to work against the original data without having to worry about degradating, you know, performance in your production system, still using production data, and then not have to worry about destructive operations as well.

• 35:41Speaker 01

  I mean, honestly, I'd love that just from a QA standpoint, right? Like, we have a tooling to easily spin up application forks to test branches or things like that. But anything data intensive is still a pain because

• 35:53Speaker 02

  Yeah.

• 35:53Speaker 01

  you gotta get a copy of your production data and you may mutate it, whatever. And that takes a lot of time in most environments today.

• 36:01Speaker 02

  And so it's all the tooling around that, which we already have the ability from the command line today to fork your database. But the challenge there, if you have a fairly large database, you have to copy that and it's going to take some time to copy that. And if you're looking at AC2 with EBS back data, it can take a while to move that stuff around and set it up. So this new fluid storage system we've been building is going to be a game changer in terms of the ability to fork much, much faster or even spin up read replicas if you want to, you needed to scale your system. For example, for peak loads where you can scale to have, you know, multiple read replicas that are load balanced behind. And you can, you know, kind of scale to peak traffic and then reduce those back down if you chose to.

• 36:43Speaker 01

  Yeah, super interesting. I think this developer experience direction is actually an interesting one to talk about. Where are the rough edges today, right? You talked about a lot of like things you want to improve there. Like, if I was coming in and using this, where am I going to get stuck as a developer right now?

• 36:58Speaker 02

  It starts with getting your chunk tuning right first and foremost. And you need to get a good amount of data in before you can, you can, you know, probably get that exactly right. I think we have some smart defaults and we actually, in our cloud environment, we actually have an engine or a solution that will help recommend chunk tuning. And we want to move that back to the CLI and MCP. The other challenge oftentimes is getting continuous aggregates right. And in particular, In the physical world, you have these data patterns that are a bit different than you see in append-only type workloads. You have devices that go offline that may, in some cases, we have a ship carrier, their devices could go offline for two weeks to a month. And then all of a

• 37:43Speaker 02

  sudden, you have all of this batch of data that can go back two weeks or a month. And so you think about these materialized views, they now have to update an entire range of data over a long period of time. And that can be CPU intensive as well as disk I/O intensive. And so getting those refresh policies right, getting all those configured correctly. And one of the other features we're adding in the future is to solve this exact problem is granular refresh, which you can think about a tenant could be a device, could be a user, whatever. When you have that backfill, you have one device is only updating that device that changed and not the entire time range. The way that continuous aggregates work today is that you update that

• 38:26Speaker 02

  entire time range within the aggregates that you have. And our ability to more finely granular address that is going to be an important piece.

• 38:35Speaker 01

  Yeah, that's interesting. So when you're doing these changes and config changes, like, how hard is it to, for example, like, say, I want to change chunk size. You talked about that, like, imagining back to my old manual partitioning days, like, doing it for new data, not a problem. Migrating all the old data, extremely intensive and challenging. Do you take care of that for us? Like, what does that migration path look like?

• 38:59Speaker 02

  So when you change the chunk size, it is changing it only for new ones. But we have functions as well for splitting and combining chunks that you can call manually to do that. Now, the challenge there is that, and we're working on reducing the locking operations around this, is that you have, you know, challenges with the locking. If you have DDL or reads or other things going on during that time, you know, it can be some challenges there. But from a developer standpoint, you just call these functions to split and combine these chunks.

• 39:29Speaker 01

  Got it. I mean, the locking raises an interesting thing. Like, if I'm doing this manipulation of the underlying structure or configuration or things like that, like, what tools do I have for managing that? So like, I think about that and I'm like, oh, okay, maybe I want to maintain this as a, you know, a duplicate for a while and then wait till I'm ready and then swap over so that I don't have to deal with, I'm locking it so I can't do anything while I'm dumping this out.

• 39:55Speaker 02

  You mean at the hyper table level or individual chunks?

• 39:58Speaker 01

  Well, it could be either, honestly. I'm just thinking about, once again, like, you're automating all of these things that I used to do by hand. When I run it to something that's not working the way I like it yet, how do I dig into that? How do I, like, what are the next level debug for, okay, my chunk size isn't right, and now I'm migrating it, and I want to be able to migrate it without any downtime? Like, what can I do as an external developer?

• 40:20Speaker 02

  There's a number of observability features built in, whether, you know, starting with our console to be able to see that. We also have on Tiger Cloud exporters for a number of different third-party tools, Data Dog, Prometheus, you know, solutions like that, that provide deeper insight into exactly what's happening in the system as well. And we're continuing to expand those. If you talk about the roadmap, we have all the Postgres metrics, but even getting into fine-grained, you know, metrics around what's happening in compression policies, CAG or continuous aggregate refresh policies, being able to see what's happening really behind the scenes as well. And Postgres has a large number of metrics that you have that

• 41:03Speaker 02

  are exposed to you to really dig in to see what's happening in the system itself, as well as all of the SQL, you know, the SQL is no different. So if you're familiar with Postgres, this is not a learning curve, other than the specific areas around hyper tables and continuous aggregates.

• 41:20Speaker 01

  Right. Well, and can I, like, use Postgres to query down a level below your sort of high level of abstraction? So like, query a particular chunk or something along those lines?

• 41:30Speaker 02

  There are functions that you can find in our API docs to, you know, access specific chunks for certain operations to see what's happening below the scenes. Those APIs are provided in our API docs. So if you really want to get into the weeds and dig into exactly how some of the sausage is made, it's there. And additionally, too, it's open source product. So you can go download the source code and take a look at it and see exactly all the APIs and how it's actually constructed.

• 42:00Speaker 01

  Yeah. No, that's awesome. That's a really nice kind of continuous story. For those of us who, you know, get nervous trusting all of the different data providers out there, and I'm like, well, you work great by default, but what happens when something goes wrong? It's a nice extension of the Postgres story, right? Postgres is great for debuggability because it has all these internal tools and metrics. And so just having that there, that's great. Well, is there anything we haven't talked about yet that you think would be important to leave folks with?

• 42:27Speaker 02

  The other area we have, and I mentioned so far, two offerings. I mentioned the open source offering with TimeScaleDB, and I mentioned Tiger Cloud. And as we grow in the physical world and industrial operations, we're also building a product called TimeScaleDB Enterprise, which is an on-prem version of the open source solution that uses a number of technologies to a lot with its own console and platform to be able to stand up an HA environment with read replicas, with, you know, connection load balancing, backups, you know, automating a lot of the things that, if you were going to build yourself, you'd have to start with the open source product, add SCD, Petroni, other technologies on top of that

• 43:12Speaker 02

  to build a true HA configuration. We're providing that out of the box as well. And we also, as part of that, have something called our Postgres connector. For developers that are interested in trying out, let's say, TimeScaleDB or Tiger Cloud, you can use our Postgres connector to point it at your database, replicate the tables you want live, and it keeps it, and it will sort of go through and play back the wall and then actually write the data and then keep those in sync. You can actually test it in parallel to your existing production database. And so that provides a really easy evaluation path. Let's say you don't want

• 43:54Speaker 02

  to download and do a PG dump and PG restore and take all the time to do that. You can just set up the PG connector. That's also leveraged in this TimeScale Enterprise DB story where you have multiple levels. Like, I'll give one example. We're working with data center operators where they have, within a building, they have these different pods, and there could be a bunch of different GPUs in the area. A pod rolls up to a building, rolls up to an enterprise. You can replicate data across multiple tiers using this connector as well. So you have your operational layer all the way down to that pod where you're monitoring, and maybe you have a group that's monitoring the entire industrial operation or the entire data center. We can replicate that data up. So it's a number of, you know,

• 44:38Speaker 02

  different technologies that sit on top of TimeScaleDB, the extensibility to run it on premise with an HA configuration, as well as, you know, this tooling to make it easier to replicate data from vanilla Postgres or TimeScaleDB to a TimeScaleDB instance.

• 44:54Speaker 01

  I mean, that feels very, very nicely consistent with what you were talking about in terms of just wanting to kind of make the plumbing problem go away for folks, where, you know, this is a problem we all have if we're managing lots of data or we're looking at putting data in different places. It's like, how do I get it from here to there? How do I do that in a way that's not disrupting my operations? So, yeah, that makes a lot of sense. So if we were to circle back, if people getting started, probably just like, if I want to install this locally, what version of Postgres do I need? You know, do I need to be on one of the most modern versions of Postgres to put TimeScaleDB in or?

• 45:29Speaker 02

  We've been 15 through 18 recently. The release actually that just came out, and it's coming out on GitHub today, which is, I think it's 2.29, deprecates PG 15. So you've got 16, 17, and 18, and we'll be adding 19 soon when it's coming out. We try to keep up to date on that. So you can, you know, just Google TimeScaleDB GitHub. There's a one-line command, curl command you can do to get it running locally. You can also go to tigerdata.com, sign up for an evaluation. We have a free 30-day evaluation with a number of credits. So you can go ahead and get started and try that out. I recommend, too, using

• 46:09Speaker 02

  the tooling, the CLI and MCP as well to get started. It's going to guide you through, and you can, you know, it's going to do through migration process, guide you through getting set up and configuring it. And what I've done is I've really just documented my use case. What do I want to build in a coding agent? And then use these tools, and it's going to use the best practices that we've put together in our documentation to build out the schema, to implement the features and get you up and running. And it can be a pretty, you know, quick turnaround to be able to do that. And then load a lot of data and really put it through its paces, because I think you'd be pleasantly surprised. And then leverage the value of our column store to compress data to reduce the data footprint.

• 46:51Speaker 02

  All of those things are pretty well sort of guided through our tooling, as well as on our console as well. We have different wizards and things to walk through it. So depending on if you're a developer that lives in the command line, stay there. And we have those tools to get you there. If you're less technical and you want to use our console and go through Tiger Cloud and use that with our, you know, SaaS console, that's another path as well. So easy path to get started, but try it out. And I think people will find a lot of value in it, especially those that have the need for that time series data that's growing, which is a common, I think, common path for people.

• 47:26Speaker 01

  I love the CLI and MCP as like the first way in, right? Because increasingly, we're all using coding agents to explore all of these tools and treating that as a kind of first-class, expected approach and saying, okay, how do we cater to that? How do we make that go on the rails rather than making assumptions it shouldn't?

• 47:43Speaker 02

  And it's not even beyond, it's beyond even getting started. We're building further tooling to really debug and validate and show, you know, slow queries, things that you could optimize within a system as well. And I think I see even building, the ability to build independent agents that are monitoring your database, monitoring the system, making recommendations in real time, above and beyond just the initial development out of the box, all using the same tools behind the scenes.